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EXECUTIVE  SUMMARY 


OBJECTIVE 

The  Space  and  Naval  Warfare  Systems  Center,  San  Diego  (SSC  San  Diego)  Command  21  project 
is  directed  at  supporting  the  needs  of  senior  decision-makers  and  support  staff  in  military  command 
centers.  As  part  of  that  effort,  the  Knowledge  Web  (K-Web)  concept  and  technologies  were  devel¬ 
oped  to  support  shared  situation  awareness  (SA),  to  facilitate  group  interaction,  and  to  augment  the 
decision-making  capabilities  of  senior  staff. 

The  K-Web  concept  was  first  tested  using  a  prototype  shared  display,  called  the  Knowledge  Wall 
(K-Wall)  at  the  Global  2000  war  game.  This  prototype  was  evaluated  against  14  user  requirements 
that  were  identified  with  a  previous  cognitive  task  analysis  of  potential  K-Web  users.  The  K-Web 
products  and  tools  met  most  of  these  requirements  with  great  success.  The  results  of  the  Global  2000 
evaluation  laid  the  foundation  for  continued  development  of  the  K-Web  products  and  tools.  New  user 
requirements  and  suggestions  for  improved  features  were  addressed  with  both  hardware  and  software 
design  changes.  Prototypes  of  these  revised  designs  were  implemented  in  the  Joint  Command  Center 
of  the  Naval  War  College  (NWC)  for  the  Global  2001  war  game.  Global  2001  afforded  an  invaluable 
opportunity  to  observe  the  redesigned  K-Web  products  and  tools  undergoing  extensive  real-time 
usage  in  an  operationally  realistic  setting,  for  which  it  was  intended. 

A  usage,  utility,  and  usability  evaluation  of  the  K-Web  products  and  tools  was  conducted  unobtru¬ 
sively  during  the  game  using  automated  data  collection  methods  and  trained  observers.  The  collected 
data  lent  to  the  assessment  of  whether: 

•  Revised  features  and  functionality  of  the  K-Web  information  products  and  support  tools,  and 
modifications  to  the  display  hardware,  meet  the  information  integration,  cognitive  support, 
and  collaboration  needs  identified  by  the  previous  Global  2000  usage  evaluation. 

•  New  and  revised  K-Web  design  solutions  support  other  previously  identified  user  require¬ 
ments. 

•  Additional  K-Web  user  needs  exist. 

•  Ergonomic  problems  related  to  the  K-Web  products  and  tools  exist. 

•  Users  prefer  the  K-Web  products  and  tools  to  other  information  tools  they  have  used. 

RESULTS 

We  found  that: 

•  The  revised  “Summary  Page”  design  was  successful  at  providing  users  with  high-level 
summarized  views  of  the  operational  picture  and  a  means  to  easily  navigate  the  K-Web  to 
obtain  more  detailed  information.  The  improved  layout  and  change  alerting  functionality 
provided  in  the  revised  design  of  the  Summary  Page  proved  useful  to  the  information  pro¬ 
ducers. 

o  Information  consumers  accessed  over  31  Summary  Pages  per  hour  over  the  course  of 
the  exercise. 

o  Information  producers  included  “change  alerting  diamonds”  on  over  50%  of  the  Sum¬ 
mary  Pages  they  created. 


•  The  K-Web  displays,  and  the  “Knowledge  Wall”  and  “Knowledge  Desks”  were  successful  at 
supporting  information  integration  and  situation  awareness  for  information  consumers  and 
producers. 

o  The  revised  Overview  Page  allowed  information  consumers  to  monitor  status  across 
all  operational  areas  simultaneously  and  quickly  access  information;  consumers 
accessed  information  from  deteriorating  areas  more  than  twice  as  quickly  as  those 
that  were  improving  or  unchanged. 

o  Consumer  comments  and  behaviors  reflected  that  they  used  these  displays  as  one  of 
their  primary  tools  for  acquiring  SA  and  becoming  apprised  of  the  concerns  of  other 
players. 

•  The  newly  designed  support  software  used  to  produce  Summary  Pages  ( SumMaker )  and  to 
control  the  display  of  information  on  the  Knowledge  Wall  and  Knowledge  Desk  ( Knowledge 
Web  Viewer)  were  a  success. 

o  Interviews  with  information  consumers  and  producers  revealed  that  they  found  these 
tools  both  useful  and  easy  to  use. 

o  However,  a  need  for  improved  alerting  and  feedback  between  information  producers 
and  consumers  related  to  content  and  usage  of  information  products  was  identified. 

RECOMMENDATIONS 

The  data  and  observations  collected  during  the  Global  2001  war  game  provided  important  feed¬ 
back  pointing  to  specific  needs  that  must  be  met  by  future  Command  21  research  and  development 
efforts.  Specifically,  we  recommend  that  future  K-Web  technology  designs  provide: 

•  Improved  mechanisms  for  dissemination  of  K-Web  business  rules  to  support  optimal  usage 
of  the  K-Web  by  information  providers  and  consumers. 

•  Enhanced  cognitive  tools  to  support  attention  management  and  change  detection,  including 
those  that  tailor  change  alerting  to  meet  the  needs  of  different  users. 

•  Efficient  and  effective  methods  for  providing  feedback  between  information  consumers  and 
providers  related  to  both  the  usage  and  content  of  K-Web  information  products. 

•  Integrated  training  to  promote  a  better  understanding  of  K-Web  tool  capabilities  by  informa¬ 
tion  producers  and  consumers. 
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1.  INTRODUCTION 


1.1.  BACKGROUND 

As  technology  increases  the  speed  and  ease  of  information  exchange,  the  problems  associated 
with  information  production,  consumption,  and  management  also  increase.  Effective  command-level 
decision-making  requires  a  high  level  of  situation  awareness  (SA).  To  support  improved  SA,  infor¬ 
mation  must  be  made  available  in  a  format  that  is  meaningful  and  useful  for  application  to  real-world 
problems;  to  do  so  means  that  the  information  must  be  transformed  into  knowledge.  How  can  infor¬ 
mation  technology  be  designed  to  best  facilitate  information  production,  consumption,  and  manage¬ 
ment?  The  Space  and  Naval  Warfare  Systems  Center,  San  Diego  (SSC  San  Diego)  is  currently 
working  to  address  this  question  through  the  Command  21  project,  an  ONR-sponsored  effort  directed 
at  supporting  the  needs  of  senior  decision-makers  and  their  support  staff  in  military  command 
centers. 

1.1.1.  The  Knowledge  Web  Concept 

A  current  focus  of  the  Command  21  effort  is  the  development  of  a  concept  referred  to  as  the 
Knowledge  Web  (K-Web).  The  K-Web  concept  utilizes  Web-based  technologies  to  share  operation¬ 
ally  relevant  information.  In  a  K-Web,  available  data  are  processed,  formatted,  and  stored  by  “infor¬ 
mation  producers”  in  such  a  way  that  it  represents  meaningful  knowledge  to  the  “information  con¬ 
sumers.”  The  “users”  of  the  K-Web,  therefore  include  both  information  producers  and  consumers. 

Prior  to  the  K-Web’ s  initial  implementation,  specific  user  information  needs  were  determined 
through  a  series  of  structured  interviews  conducted  with  Joint  Operational  Center  senior  staff  (Moore 
&  Averett,  1999;  Smallman,  Oonk,  &  Moore,  2001).  Interviews  focused  on  decision-makers’  infor¬ 
mation  requirements  in  the  context  of  an  operational  mission.  Analyses  of  the  responses  led  to  the 
identification  of  14  critical  user  requirements  that  a  tactical/operational  information  system  (e.g.,  a 
K-Web)  must  meet.  These  user  requirements  are  shown  in  Table  1. 
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Table  1.  The  14  K-Web  user  requirements. 


wsmmm 

fill  |  n  jpn  | 

_  .  Shared  SA 

Hpnpr^l  _ 

Shared  display 

Integrated  Information 

Co-located  Summary  Pages 

Intuitive  Graphical 

Interface 

Fnrmnt 

Graphical  presentation  when 
possible 

Consistency 

Consistent  format  Summary 

Pages 

Tactical  Focus 

Ability  to  view  multiple  tactical 
displays 

Supplemental  Information 

Summary  Pages  on  peripheral 
displays 

Mission  Goals  and 

Content  Objectives 

Text  document 

Anchor  Desk  Output 

Summary  Pages  with  links  to 
more  info 

Connectivity/Collaboration 

Collaboration  tools  (IWS) 

Cognitive  Support 

Limited  output  from  nonintegrated 
cognitive  support  tools 

Flexible  Configuration 

Any  pages  viewable  in  any  display 

Drill-Down 

FMtnrp  - 

Multiple  scalable  views,  links  to 
more  info 

Info  Age  and  Reliability 

- 

Tactical  Overlays 

Various  software  for  tactical 
graphic  presentation 

Based  on  these  14  user  requirements,  a  number  of  prototype  K-Web  tools  have  been  developed  to 
facilitate  the  production,  formatting,  dissemination,  and  Web-based  presentation  of  information  so 
that  it  is  easily  shared  and  understood.  These  tools  allow  information  products  to  be  created  and  con¬ 
tinually  updated  by  information  producers.  Information  products,  which  are  stored  in  shared  loca¬ 
tions  accessible  to  information  consumers  via  a  standard  Web  browser,  comprise  the  K-Web.  Other 
K-Web  tools  allow  consumers  to  easily  access  and  flexibly  configure  the  display  of  the  information 
products  on  large  or  multi-monitor  display  systems.  A  schematic  of  the  K-Web  concept  is  shown  in 
Figure  1.  Following  sections  describe  the  major  products  of  the  K-Web  and  the  hardware  and  soft¬ 
ware  tools  used  to  create  them. 
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Figure  1 .  Concept  diagram  of  the  K-Web. 

1.1.2.  Knowledge  Web  Summary  Pages 

Summary  Pages,  created  by  the  various  information  producers,  are  the  “entry  points”  to  the  K- 
Web.  They  function  as  high-level  views  of  the  operational  picture  consisting  of  single-page 
distillations  of  current  status  and  events,  critical  updates,  related  information,  relevant  graphics,  etc. 
These  pages  are  designed  to  enable  users  to  rapidly  gain  SA  of  the  overall  mission  status  through  a 
consistent  and  intuitive  graphical  format.  Consistency  in  the  placement  of  information  on  displays 
conforms  to  human  factors  principles  for  information  presentation  and  allows  users  to  scan  for 
information  efficiently  and  integrate  information  from  multiple  sources  (see  Mayhew,  1992; 
Wickens,  1992).  Consistent  formatting  also  allows  users  to  rapidly  acquire  and  integrate  information 
from  multiple  Summary  Pages  and  to  prioritize  knowledge  retrieval  from  the  K-Web.  An  example 
Summary  Page  is  shown  in  Figure  2.  Summary  Page  features  include: 

•  Color-coded  operational  status  information  across  three  time  periods:  Today  (within  24 
hours).  Tomorrow  (24  to  48  hours),  and  Long-Range  (more  than  48  hours)  and  three  colors: 
Green  (least  critical),  Yellow  (moderately  critical),  Red  (most  critical). 

•  Alerts,  impacts,  and  links  to  status-related  information. 

•  A  “focus  area”  for  a  picture  or  other  document  related  to  the  operational  situation. 

•  A  time  stamp  indicating  when  the  Summary  Page  was  last  updated. 

Most  items/elements  on  the  Summary  Pages  can  be  linked  to  additional  information  (using  hyper¬ 
links)  to  initiate  drill-down  to  more  detailed  information.  Information  producers  may  link  to  Web 
pages  (including  other  Summary  Pages),  MS  Office  documents,  graphics,  and  many  other  types  of 
information. 
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Figure  2.  Example  Summary  Page. 

1 .1 .3.  Knowledge  Web  Viewer  Hardware 

Hardware  displays  used  to  access  the  K-Web  are  called  Knowledge  Web  Viewers  (KWVs).  Spe¬ 
cifically,  the  Knowledge  Wall  (K-Wall)  used  at  Global  2001  is  a  wall-sized  shared  display  consisting 
of  contiguous  display  windows  that  bring  together  infonnation  from  multiple  sources.  The  idea 
behind  the  K-Wall  is  that  providing  a  processed  and  fused  presentation  of  the  “infonnation  space”  on 
a  single  large  display  should  support  cognitive  processes  such  as  data  integration,  pattern  recognition, 
event  memory,  and  distributed  cognition  among  infonnation  consumers.  The  K-Wall  is  intended  to 
be  large  enough  to  provide  a  shared  operational  picture  to  multiple  personnel,  to  promote  shared  SA, 
and  to  provide  focus  for  collaboration  among  its  users  (Bolstad  &  Endsley,  1999).  Smaller  single- 
user  displays  called  Knowledge  Desks  (K-Desks)  also  facilitate  K-Web  infonnation  consumption, 
but  have  the  added  purpose  of  information  production. 

1.1.4.  Knowledge  Web  Software  Tools 

Several  K-Web  software  tools  have  been  developed  that  enable: 

•  Information  producers  to  easily  create  and  disseminate  the  K-Web  infonnation  products. 

•  Infonnation  consumers  to  easily  access  and  configure  the  display  of  these  products. 

These  tools  were  developed  using  an  iterative  design  process,  involving  (1)  the  identification  of 
core  information  requirements  via  structured  interviews  with  subject  matter  experts  (Moore  & 
Averett,  1999;  Smallman  et  al.,  2001),  (2)  development  of  basic  design  requirements  and  storyboard 
concepts  by  human-computer  interface  designers,  (3)  review  of  these  concepts  by  subject  matter 
experts  and  selected  fleet  representatives  to  determine  how  well  they  met  the  identified  needs,  (4) 
reevaluation  and  reworking  of  these  concepts  until  they  met  with  approval  with  regard  to  function 
and  features  and  (5)  development  via  rapid  prototyping.  This  process  yielded  K-Web  tools  that  are 
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easy  to  learn  and  feature  easy-to-use  interfaces  that  facilitate  rapid  production,  dissemination,  and 
access  of  information  and  knowledge-based  products.  Features  that  are  familiar  to  most  users,  such 
as  “point-and-click”  and  “drag-and-drop,”  are  used  where  possible,  while  unnecessary  features  and 
functions  are  suppressed  or  removed.  The  K-Web  tool  set  includes  the  following  software: 

The  SumMaker  Software 

Information  producers  can  create  and  update  Summary  Pages  using  the  Summary  Maker  (Sum- 
Maker)  software  (Averett  &  Moore,  2000;  Averett  &  Moore,  2001).  This  software  was  developed  to 
operationalize  the  information  template  approach  adopted  by  the  Command  21  K-Web  effort.  The 
SumMaker  template-based  information  products,  and  the  SumMaker  tool  itself,  were  judged 
extremely  valuable  during  use  at  the  Global  2000  war  game  and  aboard  ship. 

Using  SumMaker ,  information  producers  with  no  knowledge  of  publishing  HTML  can  create  Web 
content.  They  can  easily  indicate  status  information  (shape  and  color)  using  pull-down  menus  and 
can  add  hyperlinks  to  most  fields  of  the  template.  Once  published,  the  HTML-based  Summary  Pages 
are  made  available  on  the  K-Web  using  standard  Web-hosting  software  (e.g.,  Microsoft’s  Internet 
Information  Server ).  Because  of  SumMaker’’ s  inherent  flexibility  to  display  or  link  to  virtually  any 
kind  of  information  product,  SumMaker  affords  the  user  the  ability  to  provide  all  relevant  informa¬ 
tion  associated  with  each  of  the  template  information  requirements.  The  SumMaker  Summary  Page 
template  is  shown  in  Figure  3. 
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Figure  3.  The  revised  SumMaker  interface. 


The  TacGraph  Software 

The  TacGraph  tool  (Bank  &  Moore,  2000;  Quinn,  Bank,  &  Moore,  2001),  allows  information 
producers  to  create  interactive  tactical  pictures  that  also  provide  value-added  information  and  hyper¬ 
links  to  other  documents.  An  advantage  of  TacGraph  over  currently  used  tools  is  that  it  enables 
multi-layered  tactical  graphics  to  be  published  in  HTML  format  and  later  edited  as  needed  (see 
Figure  4).  This  allows  graphics-based  information  products  to  be  quickly  and  easily  developed, 
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saving  considerable  time  over  the  older,  less  efficient  process  of  creating  single-use,  single-layered 
custom  tactical  graphics  with  a  combination  of  nonintegrated  applications  (e.g.,  tactical  software, 
various  photo/art/drawing  applications,  and  MS  PowerPoint). 


Figure  4.  Example  output  created  with  the  TacGraph  tool. 

The  Knowledge  Web  Viewer  Software 

Information  consumers  can  easily  access  and  configure  the  display  of  K-Web  information  products 
using  Knowledge  Web  Viewer  ( KWV)  software  (Moore  &  Averett,  2000).  KWV  software  is  a  special- 
purpose  Web  browser  that  enables  display  and  control  of  multiple  browser  windows  on  multiple 
monitors  and  facilitates  navigation  and  display  of  information  and  knowledge  residing  in  the  K-Web. 
It  also  affords  display  of  non-Web  content,  such  as  MS  Office  software  and  products.  KWV  software 
features  a  simplified  set  of  familiar  Web  browsing  controls  that  allow  easy  and  rapid  access  to  K- 
Web  content.  The  software  can  be  run  on  single-display  computer  systems  but  is  optimized  for  multi¬ 
display  systems  (up  to  six  monitors).  Figure  5  shows  the  KWV  software  running  on  two  displays. 
KWV  software  may  be  used  on  wall-sized  shared  displays,  (called  K-Walls),  as  well  as  on  smaller, 
desk-sized  units,  referred  to  as  Knowledge  Desks  (K-Desks).  Both  display  types  were  implemented 
during  Global  2001. 
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Figure  5.  The  Knowledge  Web  Viewer  (two  windows). 

The  KWV  “Overview  Page,”  shown  on  the  left  screen  in  Figure  5,  is  both  a  shared  SA  tool  and  a 
K-Web  navigation  tool.  In  order  to  enhance  shared  SA,  the  Overview  Page  combines  the  color-coded 
status  information  from  all  Summary  Pages  into  a  comprehensive  status  bar  display.  This  enables 
consumers  to  view  and  quickly  integrate  information  “at  a  glance”  and  thus  maintain  SA  regarding 
mission  status.  Navigation  features  of  the  Overview  Page  allow  users  to  easily  access  information  in 
the  K-Web.  These  features  include: 

•  Easy  access  to  Summary  Pages.  Each  button  on  the  Overview  Page  is  associated  with  one  of 
the  primary  operational  cells  that  publish  information  to  the  K-Web.  These  buttons  allow 
users  to  easily  access  a  high-level  status  summary  associated  with  each  functional  area  by 
left-clicking  with  the  mouse  on  the  appropriate  button.  If  users  want  to  see  further  informa¬ 
tion,  they  can  easily  display  the  entire  Summary  Page  (and  drill  down  as  desired)  on  any 
other  KWV  window  by  right-clicking  on  one  of  the  buttons. 

•  Text  Color  Change.  The  color  of  the  button  text  indicates  which  operational  cells  have  pub¬ 
lished  information  since  the  KWV  user  last  accessed  that  Summary  Page. 

1.1.5.  Knowledge  Web  Use  Evaluation  at  Global  2000 

The  first  in-depth  evaluation  of  the  K-Web  concept  and  tools  was  at  the  Global  2000  war  game — 
at  the  Naval  War  College  (NWC)  and  onboard  USS  Coronado  (AGF  11).  Specifically,  K-Wall 
prototypes  were  installed  for  use  by  senior  staff  (the  Commander  Joint  Task  Force  [CJTF]  staff  at  the 
NWC  and  the  Commander  in  Chief  [CINC]  staff  on  Coronado ).  Information  producers  in  each  of  13 
“functional  areas”  (the  operational  cells  designated  to  provide  information  to  the  CJTF  staff)  were 
provided  with  SumMaker  software,  which  allowed  them  to  create  Summary  Pages  for  display  on  the 
K-Wall.  K-Wall  users  could  then  manipulate  the  information  presented  on  the  K-Wall  by  using  the 
KWV  software. 

The  Global  2000  K-Web  evaluation  concluded  that  the  K-Web  successfully  supported  most  user 
requirements  and  that  users  preferred  it  to  other  tools  that  were  available  to  them  (Oonk,  Smallman, 
&  Moore,  2001).  The  K-Web  Summary  Pages  provided  consumers  with  an  integrated  summary  of 
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the  operational  picture  and  enabled  them  to  navigate  the  K-Web  in  order  to  access  further  detailed 
information.  Several  additional  requirements,  however,  were  identified  as  needing  to  be  addressed  by 
future  technology  designs.  These  included: 

•  Cognitive  tools  to  support  attention  management  and  change  detection,  including  tools 
facilitating  navigation  to  changed  pages  in  the  K-Web. 

•  Tools  and  improved  business  processes  to  support  multi-tiered  collaboration,  including  feed¬ 
back  and  guidance  from  information  consumers  to  information  producers  regarding  the  con¬ 
tent  and  use  of  K-Web  products. 

•  Improved  text  visibility. 

•  Information  integration  across  functional  areas. 

•  Information  age  and  source  information  on  the  Summary  Pages. 

As  a  result  of  this  evaluation,  changes  to  the  design  of  the  K-Web  information  products  and  tools 
were  made  to  meet  these  requirements.  These  changes  are  outlined  below. 

1.2.  REVISED  KNOWLEDGE  WEB  PRODUCTS  AND  TOOLS 

To  address  the  requirements  of  K-Web  users  identified  by  the  Global  2000  use  evaluation,  several 
revisions  were  made  to  the  design  of  the  K-Web  information  products  and  tools. 

1.2.1.  Revised  Summary  Page  Design 

The  primary  K-Web  information  products  used  during  both  Global  2000  and  Global  2001  were 
Summary  Pages.  Figure  6  shows  the  Summary  Page  design  that  was  used  in  Global  2000  and  the 
revised  Summary  Page  design  implemented  at  Global  2001. 
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Figure  6.  The  Summary  Page  design  used  in  Global  2000  (left)  and  the  revised  Summary 
Page  design  (right)  used  in  Global  2001. 

The  changes  made  to  the  Summary  Page  design  include: 

•  Change  alerting.  “New”  information  could  be  indicated  on  the  Summary  Pages  by  a  change 
in  shape  (from  a  square  to  a  diamond)  of  the  color-coded  status  indicators.  This  design 
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implementation  was  intended  to  address  the  attention  management  and  change  detection 
needs  of  the  Summary  Page  consumers. 

•  Feedback  to  information  producers  from  consumers.  The  revised  Summary  Page  layout 
included  a  field  for  the  SumMaker  user  to  provide  an  e-mail  address,  enabling  the  KWV  users 
to  contact  them  if  necessary  with  input/feedback  regarding  their  information  product. 

•  Combined  layout.  The  Alerts  and  Impacts  areas  were  combined  into  a  single  area.  These 
areas  were  combined  to  reduce  the  redundancy  that  was  observed  in  the  Summary  Page  con¬ 
tent  of  these  two  areas  during  Global  2000. 


1.2.2.  Revised  Knowledge  Web  Viewer  Hardware  Configurations 

The  initial  KWV  prototype,  the  K-Wall,  consisted  of  two  large  central  displays  surrounded  by  10 
smaller  peripheral  monitors  (see  Figure  7).  Although  this  design  provided  multiple  users  the  ability  to 
display  information  from  multiple  sources  simultaneously,  they  found  it  difficult  to  read  information 
on  the  small  monitors.  At  Global  2001,  a  large  three-monitor  configuration  replaced  the  K-Wall 
described  above.  In  addition  to  the  K-Wall,  smaller,  six-monitor  K-Desks  were  provided  to  the  vari¬ 
ous  Component  Commanders  for  use  in  information  production  and  monitoring.  Users  viewing  the 
smaller  monitors  were  now  seated  close  enough  to  permit  easy  viewing,  unlike  in  the  original  K-Wall 
configuration  used  during  Global  2000. 


K»Wail  Global  2000 


K-Wall  Global  2001  K-Desk  Global  2001 


Figure  7.  The  KWV  hardware  implemented  at  the  Global  2000  and  2001  war  games. 


1.2.3.  Revised  SumMaker  Software  Design 

The  SumMaker  software  (Averett  &  Moore,  2001)  was  revised  to  provide  additional  K-Web  func¬ 
tionality  and  to  support  the  Summary  Page  revisions  described  above  (see  Figure  8). 
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Figure  8.  SumMaker  templates. 

Three  new  functions  added  to  SumMaker  were: 

•  Feedback  to  information  producers  from  KWV  users.  The  SumMaker  software  allowed  its 
users  to  designate  the  operator  of  a  particular  KWV  as  a  “primary  watchstander.”  Feedback 
was  provided  to  information  producers  when  the  watchstander  viewed  their  Summary  Pages 
using  the  KWV  software.  This  feedback  was  provided  in  the  form  of  a  text  time  stamp  on  the 
SumMaker  software  interface.  The  purpose  of  providing  this  feedback  to  the  information 
producers  was  twofold:  (1)  to  give  them  information  about  access  and  use  of  their 
information  products,  and  (2)  to  support  the  change  alerting  functionality  (described  in 
Section  1.2.4  below). 

•  User-defined  update  rate.  Users  of  the  SumMaker  software  could  select,  using  a  pull-down 
menu,  the  rate  at  which  their  Summary  Pages  would  be  updated  (auto-refreshed)  on  the  des¬ 
ignated  watchstander’ s  KWV.  This  update  rate  can  be  set  at  various  pre-assigned  values 
(ranging  from  every  minute  to  once  every  3  hours)  based  on  how  often  the  information 
producer  believes  the  watchstander  needs  to  be  updated.  (Note  that  the  Summary  Pages  are 
updated  in  the  K-Web  every  time  a  SumMaker  information  producer  publishes  a  new  page  or 
revised  page.  The  user-defined  update  rate  only  applies  to  the  auto-refreshing  of  the  Sum¬ 
mary  Page  if  it  is  displayed  continuously  on  a  KWV). 

•  Increased  Link  capacity.  Two  additional  fields  enabled  producers  to  link  more  information  to 
the  Summary  Pages.  One  was  added  to  “related  Info  and  Links,”  and  the  other  was  added  to 
the  combined  “Alerts  and  Impacts.”  The  SumMaker  template  allowed  for  up  to  15  links  (one 
in  each  of  the  three  status  areas,  a  graphic,  seven  Alerts  and  Impacts,  and  four  Related  Infor¬ 
mation  links). 

1.2.4.  Revised  Knowledge  Web  Viewer  Software  Design 

Several  changes  were  also  incorporated  into  the  KWV  software  design  (Moore  &  Averett,  2001). 

These  changes  were: 

•  Dedicated  Overview  Page  with  support  for  Attention  Management  and  Information  Integra¬ 
tion.  Users  at  Global  2000  had  difficulty  integrating  high-level  status  information  displayed 
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on  the  K-Wall  and  keeping  track  of  changes  in  this  information.  Several  features  of  the 
revised  Overview  Page  on  Window  1  (the  top-left  window  of  the  K-Desks,  the  leftmost  win¬ 
dow  on  the  K-Wall)  are  intended  to  alleviate  these  difficulties.  The  Overview  Page  is  shown 
in  Figure  9. 
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Figure  9.  Overview  Page  designs. 

Revised  features  of  the  Overview  Page  include: 

•  Improved  overall  status  indicators  (all  areas,  over  time).  The  left  side  of  the  Global  2001 
Overview  Page  consists  of  a  status  bar  that  shows  the  status  colors  and  shapes  corresponding 
to  those  on  the  Summary  Pages  of  the  primary  operational  cells.  The  predicted  status  over 
three  time  periods — Today  (24  hours),  Tomorrow  (24  to  48  hours),  and  Long-Range  (beyond 
48  hours)  — is  displayed  in  a  single  representation  for  each  operational  cell.  The  color  (green, 
yellow,  or  red,  in  order  of  increasing  criticality)  and  shape  (square  or  diamond,  in  order  of 
increasing  recency)  convey,  respectively,  the  criticality  and  recency  of  the  information  asso¬ 
ciated  with  each  time  period.  This  was  intended  to  allow  KWV  users  to  easily  integrate 
information  from  these  areas  “at  a  glance.” 

•  Automatic  indication  of  new/changed  status  information.  When  an  information  producer  indi¬ 
cates  some  change  related  to  the  predicted  status  of  one  of  the  primary  functional  areas  (by 
changing  the  status  indicator  into  a  diamond  on  the  Summary  Page),  this  fact  is  automatically 
reflected  on  the  Overview  Page  status  bar  by  the  presence  of  a  diamond. 


1.3.  THE  KNOWLEDGE  WEB  AT  GLOBAL  2001 

1.3.1.  The  Global  2001  War  Game 

The  Global  2001  war  game  was  played  over  a  period  of  2  weeks  (16-27  July  2001).  The  2-week 
period — 10  working  days — was  composed  of  1  day  of  training,  6  days  of  operational  play  (~37.5 
hours  of  play,  intermingled  with  “time  jump”  seminars),  and  2  lA  days  of  post-exercise  Executive 
Sessions.  Game-play  featured  three  operational  phases:  a  pre-hostilities  phase  (Phase  1,  ~1 9  hours  of 
game-play),  a  hostilities  phase  (Phase  2,  ~7.5  hours  of  game-play)  and  a  post-hostilities  phase  (Phase 
3,  ~1 1  hours  of  game-play).  The  game’s  command  structure  was  composed  of  a  CJTF  supported  by 
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the  five  Functional  Component  Commands  (FCCs).1  (Throughout  this  report  these  areas  will  be 
encompassed  by  the  term  “operational  cells.”)  CINC  and  Higher  Authority  (HA;  the  command  at  the 
Pentagon)  roles  were  also  role-played  during  the  game.  A  mix  of  game  personnel.  War  College 
umpires  and  facilitators,  and  mentors  and  observers  from  various  commands  and  activities  filled 
these  roles. 

1.3.2.  The  Implementation  of  the  Knowledge  Web  at  Global  2001 

K-Web  information  products  and  tools  featuring  the  revised  design  features  described  above  were 
implemented  during  the  Global  2001  war  game.  Specifically,  this  implementation  included: 

•  A  K-Web  Web  site,  installed  on  NWC  Web  server.  This  K-Web  Web  site/server  acted  as  a 
repository  for  published  K-Web  products,  and  as  a  Web  server  so  that  game  players  could 
access  K-Web  content  via  a  Web  browser. 

•  Revised  Sum  Maker  software,  installed  on  multiple  production  consoles.  Information  produc¬ 
ers  used  these  consoles  to  create  Summary  Pages  and  other  information  products  stored  in  the 
K-Web.  Summary  Pages  were  created  by  each  of  the  FCCs  (JFACC,  JFLCC,  JFMCC, 
JSOTF,  JSTC),  the  CINC  staff,  the  CJTF  staff,  the  HA  staff,  as  well  as  various  “supplemental 
operational  areas”  (including  Intel  and  Blue’s  Red  Assessment  Team  [BRAT]). 

•  A  three-monitor  K-Wall,  installed  in  the  Joint  Command  Center  (JCC)  at  NWC.  The  display 
hardware  consisted  of  three  58-inch  diagonal  SmartBoard  rear  projection  displays.  The  K- 
Wall  was  intended  to  be  used  as  a  shared  display  by  the  CJTF  staff,  primarily  for  information 
consumption.  Figure  10  shows  the  CJTF  K-Wall.  The  KWV  software  was  the  primary  display 
software  used.  Additional  software  applications/tools  (e.g.,  MS  Office  software,  tactical  soft¬ 
ware)  were  also  used  and  displayed  on  the  K-Wall. 


Figure  10.  The  CJTF  K-Wall  in  the  JCC  at  the  NWC. 


1  Joint  Forces  Air  Component  Commander  (JFACC),  Joint  Forces  Land  Component  Commander  (JFLCC),  Joint 
Forces  Maritime  Component  Commander  (JFMCC),  Joint  Space  Theatre  Commander  (JSTC),  Joint  Special 
Operations  Task  Force  (JSOTF). 
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•  Five  six-monitor  K-Desks  at  the  NWC,  for  use  in  the  CINC  cell  and  the  operational  cells. 
These  were  intended  to  be  used  for  both  information  production  and  consumption.  Figure  1 1 
shows  an  example  K-Desk.  The  same  software  applications  that  were  used  on  the  K-Wall 
were  available  on  the  K-Desks.  The  display  hardware  consisted  of  K-Desk  computer  and  six 
15 -inch  diagonal  flat-panel  monitors. 

•  One  six-monitor  K-Desk  at  the  Pentagon,  for  use  by  HA. 


Figure  1 1 .  Example  6-monitor  K-Desk. 

1.3.3.  Training  and  Business  Process 

Figure  12  illustrates  the  implementation  of  the  K-Web  tools  within  the  Global  2001  command 
structure.  During  the  game,  a  relatively  small  number  of  game  participants  had  access  to  K-Desks 
and  K-Walls.  In  Figure  12,  an  icon  of  a  K-Desk  or  K-Wall  indicates  which  cell  had  such  a  system. 

A  larger  number  of  game  participants  had  access  to  SumMaker  and  TacGraph  software  tools, 
which  were  used  to  produce  information  products  that  were  stored  in  the  K-Web  on  the  Web  server. 
All  cells  had  access  to  these  tools,  as  indicated  by  icons  representing  Summary  Pages  in  the  lower 
right-hand  comer  of  each  blue  box  in  Figure  12. 

Finally,  all  game  participants  had  access  to  K-Web  information  products  via  their  Web  browsers 
(e.g.,  MS  Internet  Explorer).  Furthermore,  a  “Commanders’  Summaries”  link  in  the  NWC  War 
Game  Information  Grid  System  ( WIGS)  (see  section  2.2.3)  provided  easy  access  to  K-Web  content. 
Providing  all  game  participants  with  access  to  K-Web  content  was  intended  to  allow  a  more  fully  dis¬ 
tributed  set  of  knowledge/information  products  than  more  traditional  means  of  information  dissemi¬ 
nation  (i.e.,  up  and  down  the  chain  of  command).  Information  producers  were  told  that  every  partici¬ 
pant  of  the  exercise  could,  and  most  likely  would,  access  their  information  products.  However,  they 
were  also  instructed  to  create  products  with  the  Global  2001  command  structure  in  mind.2 


2  Ideally,  the  K-Web  should  be  populated  with  information  products  that  can  be  tailored  to  the  information 
requirements  of  all  users. 
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Figure  12.  The  implementation  of  the  K-Web  tools  within  the 
Global  2001  command  structure. 

Green  arrows  (the  wider  arrows)  in  Figure  12  indicate  intended  K-Web  information  flow  for 
Global  2001.  As  shown  in  Figure  12,  Summary  Pages  created  by  the  operational  cells  and  Supple¬ 
mental  Areas  were  primarily  intended  for  use  by  the  CJTF  and  his  staff.  The  CJTF  staff  created 
information  products  for  consumption  by  the  CINC  staff.  The  CINC  cell  produced  information  for 
HA.  In  this  information  flow  model,  each  echelon  primarily  “pushes”  information  up  the  hierarchy 
though  users  at  any  level  of  command  can,  and  frequently  do,  access  pages  not  specifically  designed 
for  them. 

Training  documents  (hard  copies)  were  provided  to  the  all  of  the  K-Web  tool  users.  K-Web  train¬ 
ing  documents  were  also  made  available  on-line  to  all  game  participants  via  the  K-Web  and  WIGS 
systems.  All  K-Web  training  documents  were  made  available  prior  to  game  start  and  were  available 
continuously  throughout  the  game.  The  training  documents  included: 

•  A  K-Web  business  rules  brief  (Pacific  Science  and  Engineering  Group,  2000,  2001),  which 
described  the  K-Web  concept  and  tools  and  provided  business  rules  for  K-Web  information 
products  including  those  related  to: 

o  The  intended  use  of  the  Summary  Pages  and  the  relationship  between  the  Summary 
Pages  and  the  other  K-Web  tools  (e.g.,  information  on  the  Summary  Pages  is 
reflected  on  the  KWV  Overview  Page). 

o  Use  of  color  and  shape  (change  alerting)  on  the  Summary  Pages. 

o  Appropriate  update  rates  for  information  products. 
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o  Content  and  format  of  linked  information,  including  guidelines  related  to  providing 
useful  information,  cross-referencing  others  information,  providing  legible 
information,  etc. 

o  Feedback  between  information  consumers  and  information  producers, 
o  Saving  and  accessing  information  products. 

•  A  SumMaker  training  brief  and  accompanying  “quick  reference  guide”  (Pacific  Science  and 
Engineering  Group,  2000,  2001 ),  which  provided  operator  instructions  for  users  of  the  Sum- 
Maker  software. 

•  A  KWV  training  brief  and  accompanying  “quick  reference  guide”  (Pacific  Science  and 
Engineering  Group,  2000,  2001 ),  which  provided  operator  instructions  for  users  of  the  KWV 
software. 

•  A  TacGraph  training  brief  and  accompanying  “quick  reference  guide”  (Pacific  Science  and 
Engineering  Group,  2000,  2001 ),  which  provided  operator  instructions  for  users  of  the  Tac¬ 
Graph  software. 
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2.  DATA  COLLECTION 


2.1.  PURPOSE 

The  purpose  of  the  data  collection  for  the  Global  2001  war  game  was  to  evaluate  the  use,  utility, 
and  usability  of  the  K-Web  concept  and  tools.  This  was  accomplished  by  examining  the  K-Web 
information  products  (Summary  Pages),  the  KWVs  (K-Wall  and  K-Desks)  and  the  K-Web  support 
tools  (i SumMaker  and  KWV  software)  while  they  were  undergoing  extensive  operational  use  during 
Global  2001.  The  K-Web  Global  2001  data  collection  plan  had  five  objectives: 

1 .  Verify  that  K-Web  information  products,  the  KWVs  and  the  support  tools  design  solutions 
supported  other  previously  identified  user  requirements  (see  Table  1). 

2.  Determine  whether  revised  features  and  functions  of  the  K-Web  information  products  and  the 
K-Web  support  tools,  and  the  KWVs  (K-Wall  and  K-Desks),  met  the  information  integration, 
cognitive  support,  and  collaboration  needs  identified  by  the  Global  2000  use  evaluation. 
Table  2  shows  the  revised  and  new  K-Web  features  designed  to  meet  those  needs. 

3.  Identify  whether  additional  K-Web  user  needs  exist. 

4.  Identify  ergonomic  problems  related  to  the  K-Web  products  and  tools  that  may  exist. 

5.  Assess  users’  preference  for  K-Web  products  and  tools,  relative  to  other  information  technol¬ 
ogy  products  they  have  used. 


Table  2.  The  revised  and  new  K-Web  features  implemented  and  evaluated  at  the 
Global  2001  war  game. 
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2.2.  DATA  COLLECTION  TECHNIQUES 


Data  were  collected  during  periods  of  game-play  at  the  Global  2001  war  game,  which  took  place 
from  16-27  July  2001.  Data  collection  focused  primarily  on  the  JCC  KWV  used  by  the  CJTF  staff 
and  its  users.  (Most  information  producers  published  information  primarily  for  the  JCC  KWV,  indi¬ 
cating  that  this  KWV’s  user  was  the  “primary  watchstander”).  However,  data  were  also  collected  on 
the  use  of  the  Summary  Pages  by  other  information  consumers,  and  on  the  K-Desks  and  the  Sum- 
Maker  software,  where  it  was  available. 

Data  collection  was  planned  with  flexibility  and  breadth  in  mind.  The  plan  was  flexible  to  capital¬ 
ize  on  opportunities  to  gather  relevant  data,  such  as  pertinent  verbal  communication  and  additional 
comments  that  shed  insight  into  the  cognitive  processes  underlying  use  of  the  KWV.  In  addition,  the 
flexibility  of  the  data  collection  plan  enabled  observers  to  change  or  redirect  data  collection  efforts  as 
needed.  The  data  collection  plan  was  large  in  breadth,  to  ensure  that  as  wide  a  range  of  data  was 
obtained  as  possible,  from  recording  expressions  of  preference  to  automatically  recording  keystroke 
data.  The  breadth  of  the  plan  also  ensured  that  as  much  pertinent  information  as  possible  was  col¬ 
lected  for  use  in  the  evaluation  of  the  K-Web  concept  and  technologies.  This  approach  proved  highly 
successful  at  Global  2000  and  was,  therefore,  repeated  at  Global  2001. 

To  fully  understand  the  nature  of  the  K-Web-related  data  collection,  it  is  necessary  to  first  under¬ 
stand  the  nature  of  the  environment  in  which  the  data  were  collected.  The  war  game  is  conducted  in 
the  context  of  a  free  flowing,  highly  realistic — and  therefore  unpredictable — operational  mission 
featuring  a  busy  and  intense  decision-making  environment  that  is  not  conducive  to  formal 
experimentation.  Further,  to  limit  the  impact  on  game-play,  War  College  officials  insisted  that  the 
data  collection  techniques  employed  be  primarily  restricted  to  passive  observation  and  automated 
data  collection.  The  only  exceptions  to  this  restriction  were  interviews  of  information  producers  that 
were  conducted  after  the  conclusion  of  the  exercise  (see  Section  2.2.3). 

2.2.1 .  Automated  Data  Collection 

Most  K-Web  data  were  collected  automatically  by  the  KWV  software.  Automated  data  collection 
was  conducted  in  order  to  track  URL  access  and  changes  in  the  configuration  of  KWV  displays.  Logs 
of  KWV  use  were  created  at  the  beginning  of  each  day  and  updated  each  time  a  change  was  made. 
These  logs  included  which  Web  pages  were  presented  on  which  KWV  display  and  when  the  URLs  to 
these  pages  were  accessed.  The  logs  also  included  other  keystroke  data,  such  as  when  a  button  asso¬ 
ciated  with  one  of  the  operational  cells  was  clicked,  or  when  that  Summary  Page  was  moved  to 
another  window. 

Although  the  log  files  are  a  good  source  of  information  about  the  users’  interaction  with  the 
system,  they  do  not  necessarily  give  a  complete  picture  of  what  information  was  displayed  on  the 
KWVs.  For  example,  a  user  may  access  a  Summary  Page  that  is  very  useful  and,  therefore,  display  it 
for  an  extended  period  of  time.  The  log  file  will  show  this  as  being  a  minimum  amount  of  activity 
because  the  page  was  only  accessed  once,  yet  it  would  be  meaningful  to  know  that  a  particular  page 
is  being  heavily  used.  Further,  the  KWV  users  may  have  run  other  software  (such  as  MS  PowerPoint ) 
in  a  separate  window  of  the  KWV;  this  behavior  would  not  be  indicated  in  the  log  files.  To  balance 
the  information  collected  from  the  log  files,  screen  captures  were  taken  of  the  CJTF  K-Wall  every 
minute  and  of  the  K-Desks  every  5  minutes.  The  screen  captures  provided  supplemental  information 
regarding  the  content  displayed  on  the  KWV  windows  and  how  that  information  was  configured. 
Additionally,  the  screen  captures  reflected  the  use  of  other  applications  being  displayed  that  were 
neither  accessed  through  the  KWV,  nor  recorded  in  the  log  files. 
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SumMaker  software  also  provided  built-in,  automated  data  collection  by  saving  a  copy  of  each 
Summary  Page  published.  These  files  provide  information  about  the  content  of  the  Summary  Pages, 
including  the  color  and  shape  selected  for  the  status  indicators,  the  text  of  the  alerts,  impacts,  and 
related  links,  and  the  URLs  of  links  on  Summary  Pages. 

2.2.2.  Observational  Data  Collection 

Observational  data  collection  focused  primarily  on  the  JCC  K-Wall  used  by  the  CJ'IT  staff.  These 
data  were  related  to  system  use  and  utility  of  the  K-Web  information  products  and  the  KWV,  and 
focused  on: 

•  The  revised  features  of  the  Summary  Pages  and  the  KWV,  and  how  well  they  met  users’ 
needs  for  integration,  change  alerting,  and  feedback  (between  K-Wall  users  and  information 
producers). 

•  Summary  Page  use  patterns ,  including  the  content  of  the  Summary  Pages  and  the  context  in 
which  Summary  Pages  were  accessed. 

•  K-Wall  and  K-Desk  use  patterns,  including  which  tools  and  URLs  KWV  users  access  and 
how  the  KWV  displays  were  configured. 

•  User  preferences,  with  respect  to  Summary  Pages  and  KWVs  and  compared  to  other  tools 
used  in  the  past. 

•  SumMaker  and  KWV  software  usability,  including  issues  related  to  the  ease  and  intuitiveness 
of  use. 

•  Suggestions  and  new  requirements,  in  terms  of  the  content,  features,  and  tools  of  the  Sum¬ 
mary  Pages  and  KWV. 

•  Visibility  and  other  ergonomic  issues  related  to  the  KWV,  including  the  legibility  of  text  and 
graphics  on  the  KWV  and  issues  associated  with  occlusions,  monitor  glare,  room  layout, 
communication  between  personnel,  etc. 

The  specific  data  sources  used  to  collect  observational  data  included: 

•  Logging  of  critical  events  that  caused  a  change  in  Summary  Page  and  KWV  use. 

•  Time  sampling  of  the  content  and  configuration  of  the  KWVs,  as  well  as  which  tools  and 
Summary  Pages  were  being  used  or  viewed  (discussed). 

•  Manually  collected  record  of  verbal  comments  made  by  KWV  users  as  they  relate  to  usability 
issues,  new  requirements,  preferences,  and  visibility. 

•  Answers  to  verbal  probing  of  specific  K-Web  users  about  the  above  issues  (new  KWV  and 
Summary  Page  features,  usability,  requirements,  preference,  and  visibility)  if  observations  of 
behavior  have  not  provided  enough  information. 

•  Measurements  and  descriptions  of  variables  related  to  ergonomic  use  and  visibility  of 
information  on  the  KWV.  Measurements  were  taken  of  the  layout  of  JCC  space.  Visibility  of 
the  information  presented  on  the  K-Wall  was  also  measured. 

The  observational  data  collection  described  above  occurred  continuously.  Similar  data  were  col¬ 
lected  across  the  K-Desk  users,  whenever  possible. 
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2.2.3.  Other  Observational  Data  Sources 


Interviews  with  SumMaker  and  KWV  Software  Users.  At  the  conclusion  of  the  exercise,  interviews 
were  conducted  with  the  information  producers  from  each  operational  cell  that  used  the  SumMaker 
software  to  create  Summary  Pages,  and  with  information  consumers  that  used  the  KWV  software 
using  the  five  K-Desks.  Nine  users  were  interviewed  in  total.  The  participants’  answers  were 
included  as  part  of  the  observational  data.  Participants  in  these  interviews  were  asked  the  following 
five  questions: 

1 .  What  do  you  think  were  the  best  features/functions  of  the  SumMaker/KWV  software? 

2.  Generally  speaking,  did  the  SumMaker/KWV  software  meet  your  needs  as  an  information 
producer? 

3.  What  features/functions  should  be  added  or  improved  to  make  the  SumMaker/KWV  software 
better  meet  your  needs? 

4.  How  could/should  the  information  production  and  dissemination  process  be  improved? 

5.  Do  you  have  any  further  comments  or  suggestions  for  us? 

Knowledge  Management  Meetings.  The  observational  data  include  comments  made  during  semi¬ 
nars,  discussions  with  the  Knowledge  Managers  (KMs),  and  the  KM  meetings  that  were  held  twice  a 
day.  The  KMs  provided  invaluable  insights  into  many  issues  related  to  the  K-Web  because  of  their 
roles  as  facilitators  for  the  use  of  the  K-Web  and  other  information  technologies.  These  perceptions 
were  also  documented  and  made  available  to  members  of  the  Command  21  program  team  during 
Global  2001. 

War  Game  Information  Grid  System  (WIGS).  Many  documents  and  tools  used  by  the  information 
consumers  could  be  accessed  via  WIGS,  a  Web-based  system  provided  by  the  NWC.  Two  of  these  in 
particular  provided  important  data  regarding  the  K-Web.  The  first  was  the  CJTF  watch  log,  main¬ 
tained  by  CJTF  staff  using  collaboration  software  (specifically  CommandNet3).  Another  source  of 
information  was  the  situation  summary  (the  SitSum)  that  was  updated  continuously  by  exercise  par¬ 
ticipants  throughout  the  exercise.  Access  to  both  sources  provided  information  about  critical  events. 


3 


http://www.cmi.arizona.edu/research/dist  collab/CommandNet/index.htm 
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3.  RESULTS 


To  test  and  evaluate  the  effectiveness  of  K-Web  tools  and  information  products,  a  comprehensive 
data  set  was  collected  from  each  Global  2001  participant  who  used  the  K-Web.  These  data  included 
both  observational  data  collected  by  the  K-Web  research  staff  and  data  gathered  automatically  by 
each  K-Web  tool.  When  considered  jointly,  these  two  types  of  data  yielded  a  thorough  description  of 
participant  use  of  the  K-Web.  Appendix  A  contains  summary  tables  of  the  data  that  were  collected 
automatically;  Appendix  B  contains  summary  tables  of  the  observational  data. 

Data  collected  automatically  by  the  SumMaker  software  containing  information  related  to  the 
content  of  Summary  Pages  published  during  the  game  were  contained  in  837  files.  Data  collected 
automatically  by  the  KWV  software  and  related  to  the  KWV  use  was  analyzed  for  three  KWV s:  the 
CJTF  K-Wall,  the  CINC  K-Desk,  and  one  of  the  FCC  K-Desks.4  The  KWV  data  included  2145  screen 
captures  taken  of  the  K-Wall  used  by  the  CJTF  staff,  423  screen  captures  taken  of  the  CINC  K-Desk, 
and  349  screen  captures  taken  of  the  FCC  K-Desk.  These  data  provided  information  about  the  con¬ 
tent  and  configuration  of  the  information  on  the  K-Wall  and  K-Desks.  Data  logs  from  the  K-Wall  and 
four  of  the  K-Desks  provided  information  related  to  the  use  of  the  KWV  software.5 

The  observational  data  included  approximately  200  comments  and  observed  behaviors  made  by 
CJTF  staff  and  other  users  throughout  the  course  of  the  exercise.  The  data  collected  automatically  by 
the  KWV  and  SumMaker  software  came  from  users  of  the  K-Web  tools  from  the  CJTF  staff,  the 
CINC  staff,  five  FCCs  (henceforth  referred  to  as  FCC1,  FCC2,  FCC3,  FCC4,  and  FCC5)  and  two 
Supplemental  Areas  (Supp Areal  and  SuppArea2). 

Observational  data  also  included  data  related  to  critical  events.  A  Subject  Matter  Expert  (SME) 
identified  16  of  the  most  important  developments  during  the  exercise  based  upon  information  from 
CJTF  watch  logs,  the  SitSum,  and  the  critical  event  logs  from  each  day  of  the  exercise.  Three  other 
SME  raters  assigned  a  “criticality”  rating  to  these  events,  (4  =  critical  event,  3  =  significant  event, 

2  =  important  event,  1  =  event  of  interest).  The  average  critical  ratings  ranged  from  1.7  to  3.7.  These 
critical  events  were  included  in  several  of  the  analyses  discussed  below. 

Automatically  collected  and  observational  data  were  distributed  into  six  categories,  related  to: 

1 .  The  use,  utility,  and  usability  of  the  K-Web  information  products  (primarily  Summary  Pages) 
during  Global  2001. 


4  Only  data  from  a  single,  representative  K-Desk  were  included  in  the  analyses  for  a  number  of  reasons.  For 
example,  the  KWV  application  was  not  run  on  one  of  the  FCC  K-Desks  so  no  data  log  was  created  for  it.  Only  a 
limited  number  of  screen  captures  were  taken  of  this  K-Desk.  Therefore,  data  from  this  KWV  were  not  included  in 
any  analyses.  Due  to  technical  problems,  no  screen  captures  were  taken  of  another  FCC  K-Desk,  so  no  data  related 
to  this  KWV  were  included  in  the  analyses.  A  third  FCC  K-Desk  was  operated  almost  entirely  by  members  of  the 
Command  21  K-Web  support  team,  under  the  direction  of  a  civilian  exercise  participant.  We  were  concerned  that 
the  usage  patterns  of  these  operators  would  not  be  representative  of  the  usage  by  other  FCC  KWV  operators  and  so 
excluded  die  data  collected  automatically  by  the  KWV  software  from  this  K-Desk.  Note  that  the  observational  data 
regarding  these  three  KWVs  are  included  in  the  Results. 

5  The  KWV  application  was  not  run  on  one  of  the  K-Desks  (in  one  of  the  FCCs),  so  no  data  log  was  created  for  it.  A 
limited  number  of  screen  captures  were  taken  of  this  K-Desk  and,  therefore,  those  data  were  not  included  in  any  of 
the  analyses.  Further,  due  to  technical  problems,  no  screen  captures  were  taken  of  another  FCC  K-Desk.  Due  to 
these  circumstances,  screen  capture  data  were  included  for  three  of  the  five  K-Desks,  and  log  data  were  included  for 
four  of  the  five  K-Desks. 
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2.  The  revised  Summary  Page  design  features  (content,  layout,  change  alerting). 

3.  The  use,  utility,  and  usability  of  the  K-Wall  and  K-Desks  during  Global  2001 . 

4.  The  revised  K-Wall/K-Desk  design  features  (number  of  monitors,  change  alerting). 

5.  Revised  SumMaker  and  KWV  software  features  and  usability. 

6.  Other  emerging  themes,  especially  training  and  business  rules. 

The  results  are  organized  around  these  categories  and  include  discussion  of  the  observational  data, 
as  well  as  appropriate  analyses  conducted  on  the  data  collected  automatically  (i.e.,  the  KWV  use  logs 
and  screen  captures).6 

3.1.  USE,  UTILITY,  AND  USABILITY  OF  SUMMARY  PAGES  DURING  GLOBAL  2001 

3.1.1.  Summary  Page  Use 

Summary  Pages  and  the  products  that  they  linked  to  were  the  predominant  products  in  the  K-Web. 
Information  producers  used  Summary  Pages  to  update  senior  staff  and  others  about  important  issues, 
events,  and  plans,  with  respect  to  operational  play.  Summary  Pages  could  be  accessed  easily  using 
the  KWV  software  and  displayed  on  the  K-Wall  and  K-Desks. 

The  number  of  times  that  Summary  Pages  and  links  were  accessed  per  hour  of  use  on  the  K-Wall 
and  K-Desks  is  shown  in  Figure  13  and  Figure  14,  respectively.  Use  is  shown  as  a  function  of  opera¬ 
tional  phase  (see  Section  1.3.1).  Analysis  of  the  KWV  data  logs  indicated  that  Summary  Pages  were 
frequently  accessed  throughout  the  exercise.  An  average  of  27.5,  30.0,  and  38.8  Summary  Pages 
were  accessed  per  hour  in  Phases  1,  2,  and  3,  respectively,  across  the  KWVs.  The  increase  in 
Summary  Page  access  over  the  course  of  the  exercise  is  not  surprising  because,  over  time, 
information  consumers  should  become  more  familiar  with  the  display  technology  and  the 
information  producers  should  provide  more  “developed  pages.”7 

In  addition,  links  from  these  Summary  Pages  were  frequently  accessed  (an  average  of  41.5,  25.0, 
and  27.2  links  were  accessed  per  hour  in  Phases  1,  2,  and  3),  particularly  by  the  CJTF  and  CINC 
KWV  users.8  The  display  of  Summary  Pages  on  the  K-Wall  and  K-Desks  is  discussed  further  in 
Section  3.3.1. 


6  Note:  Comments  made  are  presented  in  italicized  text.  Many  of  them  are  paraphrased  or  have  been  corrected  for 
grammar.  Observations  are  presented  in  normal  font. 

7  The  difference  in  usage  patterns  (here,  and  in  subsequently  reported  results)  across  operational  areas  and  across 
exercises  is  also  not  surprising,  given  there  were  many  uncontrolled  variables  that  could  account  for  the  differences. 
These  variables  include,  but  are  not  limited  to,  organizational  variables  (differences  across  the  commands  that 
participated  in  the  game  and  across  the  organizational  structures  imposed  by  the  game)  participants”  familiarity  with 
Web  technology,  exercise  scenarios,  and  information  needs  of  the  operational  areas.  More  discussion  of  these 
differences  is  found  in  Section  3.7.1.  of  the  Results  (Training  and  Business  Rules). 

8  Note  that  these  data  do  not  include  instances  when  K-Wall/K-Desk  users  used  the  Back  button  on  the  KWV 
application  to  return  to  a  Summary  Page  after  accessing  a  link  from  that  Summary  Page. 
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Figure  13.  The  number  of  Summary  Pages  accessed  via  the 
KWV  software  on  the  K-Wall  and  K-Desks. 


Figure  14.  The  number  of  links  from  the  Summary  Pages 
accessed  via  the  KWV  software  on  the  K-Wall  and  K-Desks. 

In  addition  to  access  via  the  KWV  software,  all  information  consumers  could  view  Summary  Pages 
via  a  link  to  the  Commander’s  Summaries  on  the  War  College  Web  (WIGS).  Although  patterns  of 
access  varied,  information  consumers  in  all  cells,  from  the  FCCs  up  to  HA,  used  the  Summary  Pages 
to  become  apprised  of  the  operational  situation  and  of  other  exercise  participants’  concerns.  Obser¬ 
vations  and  comments  included: 

•  The  Commander’s  Summaries  are  accessed  by  the  cell  players  and  projected  up  on  the 
K-Wall 

•  [FCC2]  used  the  K-Desk  to  obtain  situation  awareness  (S A)  from  the  other  Summary 
Pages...  so  the  KM  could  keep  track  of  other  pages  and  to  update  his  own  Summary  Page  and 
documents. 

•  I’m  seeing  guys  dipping  down,  grabbing  pictures,  graphics,  text,  and  providing  it  to 
Commanders. 

•  In  my  cell,  5  to  6 people  at  a  time  are  looking  at  the  Commanders’  Summary  Pages. 

•  I’m  looking  at  the  pages  to  make  sure  no  information  needs  filling  in... there’s  good  informa¬ 
tion  that’s  been  converted  to  knowledge  if  you  know  where  to  get  it. 
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•  The  Commanders’  Summary  Page — gave  [FCC2]  a  way  to  give  a  picture  to  the  JTF  staff  the 
way  that  we  wanted/thought  they  should  see  it. 

3.1.2.  Summary  Page  Update  Rate 

If  the  K-Web  is  to  provide  information  consumers  with  current  and  accurate  information,  the 
Summary  Pages  must  be  updated  by  information  producers  at  an  adequate  rate  to  reflect  events  and 
changes  in  the  operational  situation.  The  rate  at  which  the  information  producers  updated  their  Sum¬ 
mary  Pages  is  shown  in  Figure  15.  Different  information  producers  published  at  different  rates  over 
the  course  of  the  exercise.  This  is  not  surprising  because  the  amount  of  information  that  is  relevant  to 
the  operational  situation  varies  at  any  point  in  time  across  the  different  operational  cells.  The  design 
of  the  SumMaker  enables  users  to  update  “as  needed,”  as  opposed  to  requiring  scheduled  periodic 
updates.  The  SumMaker  software  also  allows  users  to  select  the  rate  at  which  their  pages  are 
automatically  refreshed  on  the  KWVs  (see  section  1.2.1,  Revised  Summary  Page  Design),  thereby 
providing  the  most  accurate  and  timely  information  available  while  at  the  same  time  conserving 
bandwidth. 
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Figure  15.  The  rate  at  which  Summary  Pages  were  published  (number  of 
Summary  Pages  posted  per  hour  of  the  exercise). 

On  average,  a  Summary  Page  was  published  to  the  K-Web  quite  frequently — every  3.14  minutes 
(every  2.78  minutes  in  Phase  1,  every  2.57  minutes  in  Phase  2,  and  every  4.07  minutes  in  Phase  3). 
Note  that  this  update  rate  does  not  take  into  account  the  creation  and  revision  of  the  K-Web  informa¬ 
tion  products  that  were  not  Summary  Pages  (i.e.,  the  products  that  were  links  from  the  Summary 
Pages).  As  shown,  the  update  rates  varied  considerably  across  the  various  functional  areas,  con¬ 
firming  the  need  for  user-controlled  update  rates. 

The  information  consumers’  level  of  satisfaction  with  update  rates  varied.  Senior  staff  occasionally 
instructed  the  FCCs  (via  chat  and  e-mail)  to  provide  more  current  information  on  the  Summary 
Pages.  Related  comments  included: 

•  [ FCC5  ]  has  got  their  stuff  populated. . .  that ’s  good. 

•  There  still  isn  ’t  enough  content  on  the  Summary  Pages. 
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•  We’ve  looked  on  [FCC3]  and  [FCC2]  pages  and  can’t  find  evidence  of  [a  trigger]... We’ve 
already  sent  them  a  message  to  update  their  Commander’s  Summary  Pages. 

•  The  information  is  coming  too  fast  to  update  the  slide  (link  re:  above)  anymore... we  were 
told  to  use  the  tools  until  they  break  and  it  looks  like  they  have. 

•  We  need  all  of  you  to  update  your  Commander’s  Summary  Pages  with  information  about 
where  you  are.... 

•  We  need  you  to  update  the  Commander’s  Summaries  every  morning,  afternoon,  and  as 
needed  for  everyone... need  to  push  other  information  during  the  day. 

3.1.3.  Summary  Page  Preference 

In  general,  information  consumers  liked  the  Summary  Pages  and  some  preferred  them  as  a  means 
to  communicate  status  to  senior  staff  and  other  information  consumers,  relative  to  other  tools.  Many 
suggestions  were  made  for  improvements  for  new  features  and  tools.  These  are  discussed  below  (see 
the  discussion  of  the  Revised  Summary  Page  Design,  Section  3.2).  Representative  comments  were: 

•  The  Summary  Page  gave  [FCC2 ]  a  way  to  give  a  picture  to  the  JTF  staff  the  way  that  we 
wanted/thought  they  should  see  it. 

•  It  standardized  everyone’s  input. 

•  We  ’re  trying  to  figure  out  templates  for  our  Web  Page  on  the  ship. .  .these  Summary  Pages 
would  work. 

•  The  tools  found  to  be  very  useful  included  the  K-Wall  setup,  Commander’s  Summaries.... 

3.2.  REVISED  SUMMARY  PAGE  DESIGN 


3.2.1.  Change  Alerting 

One  new  feature  of  the  Summary  Page  design 
was  indicators  of  new  or  changed  information. 
SumMaker  gave  information  providers  the  ability  to 
indicate  changed  and  new  information  by  changing 
the  shape  (from  a  square  to  a  diamond)  of  the  color- 
coded  status  indicators.  Information  providers  could 
also  include  these  “change  diamonds”  in  the  Alerts 
and  Impacts  section  to  indicate  which  information 
was  associated  with  a  change  in  status. 
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Figure  16  shows  the  percentage  of  Summary  Pages  that  included  status  change  diamonds.  Analysis 
of  the  KWS  files  indicated  that  the  information  providers  included  change  diamonds  very  frequently 
(on  average,  in  34%,  62%,  and  70%  of  the  Summary  Pages  that  were  published  in  Phases  1,  2,  and  3, 
respectively). 
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Figure  16.  The  percentage  of  Summary  Pages  that  contained  one  or  more 
change  diamonds. 

Data  from  the  KWS  files  were  also  analyzed  to  determine  whether  change  diamonds  were  being 
used  differently  after  a  critical  event  than  during  times  when  there  was  no  critical  event.  Figure  17 
shows  the  percentage  of  published  Summary  Pages  that  contained  change  diamonds  during  the  hour 
following  critical  events  (least  critical  =  an  average  rating  of  less  than  2,  moderately  critical  =  rated 
between  2  and  3,  most  critical  =  rated  higher  than  3).  The  figure  also  shows  the  average  proportion  of 
published  Summary  Pages  that  contained  change  diamonds  when  there  was  no  critical  event.  As  the 
criticality  of  the  operational  situation  increased,  information  providers  included  more  change  dia¬ 
monds  on  the  Summary  Pages.  This  reflects  what  would  be  expected;  as  criticality  level  increases, 
more  factors  related  to  operational  status  would  likely  be  affected. 


No  Critical 
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CE  Value 


Figure  17.  The  percentage  of  published  Summary  Pages  that 
contained  change  diamonds,  as  a  function  of  level  of  criticality. 

The  discussion  of  the  revised  KWV  Overview  Page  (Section  3.6.1)  describes  in  more  detail  how 
effective  the  change  diamonds  were  in  alerting  users  of  the  K-Wall  to  new  information.  Comments 
related  to  the  summary  change  diamonds  suggested  that,  although  users  liked  them,  further  change 
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alerting  issues  remain.  Representative  comments  related  to  the  change  diamonds  on  the  Summary 
Pages  are: 

•  The  diamonds  were  good. . . . 

•  The  main  feature  that  is  needed  is  alert  functionality. 

•  How  do  you  alert,  tailor  information  for  different  CCs? 

•  You  can  post  things  on  the  Web  all  day  long  but  you  need  to  push  information... can’t  assume 
people  will  pull  off  Web. 

•  Need  an  indication  of  when  alerts  were  changed  or  added. 

•  Since  it’s  always  on  the  display. ..if  you  know  in  advance  what  you  want  to  be  alerted  to  you 
could  be  automatically  alerted.... 


3.2.2.  Revised  Layout  of  Summary  Pages 

Combined  Alerts  and  Impacts.  Another  change 
made  to  the  design  of  the  Summary  Page  was  the 
combination  of  the  Alerts  and  Impacts  areas  into  a 
single  area.  Information  providers  could  include  up 
to  seven  alerts  in  this  section  (along  with  an  associ¬ 
ated  link  and  a  colored  square  or  diamond  for  each) 
and  up  to  four  pieces  of  information  in  the  Related 
Links  section.  Each  of  these  1 1  fields  (alerts  and 
related  links)  could  have  a  hyperlink  associated  wit! 
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User  comments  indicate  a  preference  for  the  standardized  Summary  Page  format.  Some  informa¬ 
tion  providers,  however,  thought  the  Summary  Pages  were  too  simple  and  indicated  a  need  to  change 
the  design  so  that  the  content  and  quality  of  information  could  be  changed.  Comments  related  to  the 
layout  and  features  of  the  Summary  Pages  included: 


•  I  like  the  idea  of  everyone  having  information  in  the  same  format. 

•  I  need  more  space  to  put  stuff. 

•  [The  Summary  Page]  is  a  good  looking  page  with  enough  white  space. 

•  There  was  only  a  limited  amount  of  information  you  could  put  on  it. 

•  The  Summary  Pages  were  too  simplistic. 

•  It  wasn’t  flexible  enough.. .you  couldn’t  change  the  information. 

•  You  should  always  put  military  time  on  the  Summary  Pages. 

•  Nice  feature  to  have  the  time  stamp  though. 


Information  Producer  E-mail  Address.  The 
revised  Summary  Page  design  also  included  a  field 
in  which  the  information  provider  could  post  his  or 
her  e-mail  address  so  anyone  viewing  the  Summary 
Page  could  send  them  feedback.  Information  produc¬ 
ers  expressed  dissatisfaction  in  the  amount  of  feed¬ 
back  that  they  received  about  the  content  of  their 
information  products.  This  is  in  spite  of  the  fact  that 
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the  K-Wall  users  (as  well  as  the  CINC  and  HA)  were  observed  to  provide  feedback  to  these  users  in 
the  form  of  e-mail  or  other  communications.  Related  comments  included: 

•  We  need  all  of  you  to  update  your  Commanders  ’  Summary  pages  with  information  about 
where  you  are... [chat  message]. 

•  I’m  looking  at  the  pages  to  make  sure  no  information  needs  filling  in.  If  there  is,  I  tell  BW1 
and  he  lets  the  guy  know...BWI  is  pretty  proactive  about  it  too. 

•  E-mail  from  CENC  that  Today’s  link  is  still  yesterday’s  plan. 

•  We  didn’t  get  enough  feedback  from  upstairs  [JCC]  about  whether  there  was  right  informa¬ 
tion. 

•  We  need  to  know  what  the  CJTF  wants  to  see.... 


3.3.  USE,  UTILITY,  AND  USABILITY  OF  KNOWLEDGE  WEB  VIEWER  HARDWARE  DURING 
GLOBAL  2001 


3.3.1.  Format  and  Content  of  Information  on  the  Knowledge  Wall/Desks 

Analysis  of  the  screen  captures  indicated  that  the  format  of  the  information  displayed  on  the  win¬ 
dows  varied  across  the  different  K-Walls/K-Desks.  The  dedicated  Overview  Page  was  always  dis¬ 
played  on  Window  1  of  the  CJTF  K-Wall;  therefore,  the  format  of  the  information  there  never 
changed.  Figure  18  shows  the  percentage  of  time  information  was  displayed  on  the  other  windows  of 
the  K-Wall  as  a  function  of  operational  phase  and  format  of  the  information  displayed  (averaged 
across  Windows  2  and  3).  Figure  19  and  Figure  20  show  the  percentage  of  time  information  was 
displayed  on  the  respective  K-Desks  as  a  function  of  operational  phase  and  format  of  the  information 
displayed  (averaged  across  all  six  K-Desk  screens). 

“Auxiliary”  K-Web  products  were  any  products  found  in  the  K-Web  that  were  not  Summary 
Pages,  and  were  usually  a  link  from  a  Summary  Page.  As  can  be  seen  in  Figures  18  through  20,  most 
of  information  products  that  were  displayed  were  auxiliary  K-Web  products,  primarily  displayed  in 
Window  2  of  the  K-Wall,  and  WIGS  products,  primarily  displayed  in  Window  3.  Either  a  Summary 
Page  or  an  auxiliary  K-Web  product  was  displayed  on  each  monitor  of  the  K-Wall  58.5%  of  the  time 
(Summary  Pages  were  displayed  5%  and  auxiliary  K-Web  products  were  displayed  53.5%  of  the 
time  averaging  across  Windows  2  and  3).  MS  PowerPoint  products  were  also  frequently  displayed, 
usually  during  the  morning  and  afternoon  CJTF  briefings.  The  information  products  displayed  on  the 
CINC  and  FCC  K-Desks  were  more  varied,  but  K-Web  products  still  composed  the  most  commonly 
displayed  information.  K-Web  products  were  displayed  on  each  monitor  of  the  CINC  K-Desks 
45.4%  of  the  time  (Summary  Pages  were  displayed  24.3%,  auxiliary  K-Web  products  12.1%,  and  the 
Overview  Page  was  displayed  9%  of  the  time  averaging  across  all  six  monitors)  and  on  the  FCC  K- 
Desk  23.9%  of  the  time  (Summary  Pages  were  displayed  14.2%,  auxiliary  K-Web  products  3.2%, 
and  the  Overview  Page  was  displayed  6.5%  of  the  time). 
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Figure  18.  The  format  of  the  information  displayed  on  Windows  2  and  3  of  the  K-Wall. 
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Figure  19.  The  format  of  the  information  displayed  on  the  CINC  K-Desk.9 


9  SP  =  Summary  Page,  Aux.  KWeb  =  auxiliary  K-Web  (link),  OvView  =  overview  page  in  WIGS  (link  to 
Commanders  Summary  Page),  NWC  Web  =  WIGS,  MS  Off  =  MS  Office  products,  Tact  =  tactical 
information,  SmMker  =  SumMaker  software,  TcGrph  =  TacGraph  software. 


29 


Figure  20.  The  format  of  the  information  displayed  on  the  FCC  K-Desk. 

Figure  21,  Figure  22,  and  Figure  23  show  the  content  of  the  information  displayed  on  the  K-Wall 
and  K-Desks,  as  indicated  by  the  screen  captures.  Most  of  the  K-Web  products  displayed  on  the 
CJTF  K-Wall  were  those  created  by  the  Supplemental  Areas  (displayed  42.1%  of  the  time),  whereas 
the  FCC  K-Desk  was  more  likely  to  display  Summary  Pages  and  links  from  the  other  FCCs  (44.6% 
of  the  time).  Not  surprisingly,  the  CINC  K-Desk  frequently  displayed  information  from  the  more 
senior  commanders  (the  CJTF,  the  CINC  and  HA)  as  well  as  from  the  FCCs  (39.0%  and  33.7%  of 
the  time,  respectively).  The  K-Wall  and  FCC  K-Desk  frequently  displayed  the  CommandNet  soft¬ 
ware  (44.6%  and  29.5%  of  the  time,  respectively),  available  through  the  WIGS  system  to  maintain 
watch  logs.  The  CINC  K-Desk  also  frequently  displayed  the  output  from  Theater  Assessment  Profil¬ 
ing  System  (TAPS),  which  provided  effects-based  assessment  of  the  operational  system  and  was 
available  through  WIGS  (16.8%  of  the  time). 


Comp.  Cdr.  SuppArea  CJTF,  CINC,  CmdNet  SitSum  TAPS 

HA 

Content 


Figure  21 .  Content  of  the  information  displayed  on  Windows  2  and  3  of  the  K-Wall.10 


10  Comp.  Cdr.  =  FCCs,  SuppArea  =  Supplemental  Area,  Cdr.  Sum  =  WIGS  access  to  the  operational  cell  Summary 
Pages,  CmdNet  =  CommandNet  (via  WIGS),  TAPS  =  Theater  Assessment  Profiling  System. 
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Figure  22.  Content  of  the  information  displayed  on  the  CINC  K-Desk. 
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Figure  23.  Content  of  the  information  displayed  on  the  FCC  K-Desk. 


3.3.2.  Knowledge  Wall/Desk  Use 

Differences  in  the  format  and  content  across  the  K-Wall  and  K-Desks  provided  evidence  that  dif¬ 
ferent  users  and  organizational  cells  used  the  KWVs  in  diverse  ways.  Observed  behaviors  and  com¬ 
ments  made  by  users  of  the  JCC  K-Wall  indicated  that  the  CJTF  staff  used  the  information  displayed 
to  maintain  SA  and  to  stay  apprised  of  the  concerns  and  intentions  of  the  FCCs,  the  CINC,  and  HA, 
and,  especially,  the  Supplemental  Areas.  The  K-Wall  was  also  used  to  maintain  and  display  the  CJTF 
watch  log  (in  CommandNet),  as  well  as  for  the  morning  and  afternoon  CJTF  briefings,  and  for 
smaller,  event-related  briefs  throughout  each  day. 

In  addition  to  using  K-Desks  for  maintaining  SA,  the  FCCs  also  used  K-Desks  to  create  and  edit 
information  products  (Summary  Pages,  MS  Office  documents),  to  collaborate  with  others  (via  e-mail 
and  chat),  and  to  maintain  watch  logs.  The  frequency  and  sophistication  of  K-Desk  use  varied  across 
organizational  cells,  ranging  from  continuous  to  occasional,  and  simple  to  complex  usage  patterns. 
For  example,  the  K-Desk  users  in  one  FCC  typically  displayed  an  Overview  Page  and  a  couple  of 
Summary  Pages  on  Windows  1  through  3  for  the  majority  of  the  game,  but  they  did  not  drill  down  or 
navigate  through  the  K-Web.  Another  FCC  continuously  changed  the  Summary  Pages  being  viewed 
and  navigated  links  for  further  information.  In  several  FCCs,  one  or  more  of  the  K-Desk  displays 
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were  projected  onto  a  large-screen  display  (at  the  front  of  the  room)  so  that  information  on  those 
windows  could  be  shared  and  viewed  by  all  information  consumers  in  that  FCC. 

The  log  files  provided  some  insight  as  to  how  the  KWVs  were  being  used  at  different  points  in  the 
game.  These  data  were  analyzed  to  determine  if  there  was  a  change  in  activity  level  during  the  20 
minutes  prior  to,  and  30  minutes  following  each  critical  event."  Activity  was  defined  as  any  mouse 
action  by  the  user  (e.g.  right  or  left  click)  from  a  KWV.  Figure  24  represents  the  activity  (across  the 
CINC,  CJTF,  and  FCC  KWVs)  surrounding  critical  events,  as  a  function  of  the  level  of  criticality. 
The  results  show  that  the  activity  before  critical  events  was  varied,  depending  on  the  level  of  critical¬ 
ity,  with  the  least  critical  events  having  the  most  activity  and  the  most  critical  events  having  the  least 
activity.  This  is  difficult  to  interpret,  but  may  be  due  to  differences  in  the  content  of  the  K-Web  prior 
to  critical  events.  User  activity  after  the  critical  events  was  virtually  the  same  for  all  levels  of  criti¬ 
cality  and  consistently  showed  a  decrease  in  activity  following  a  critical  event.  This  decrease  may  be 
attributed  to  users  focusing  on,  and  carefully  considering,  information  associated  with  the  critical 
event  that  occurred  rather  than  continuing  to  gather  information  from  sources  unrelated  to  the  event. 
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Figure  24.  The  activity  (across  the  CINC,  CJTF  and  FCC  KWVs) 
surrounding  critical  events,  as  a  function  of  the  level  of  criticality. 

Comments  related  to  KWV  use  are  as  follows: 

K-Wall  Use 

•  People  in  the  JCC  completely  relied  on  the  K-Wall. 

•  ... we  can  put  whatever  we  want  up  there. ..The  BWCs  are  on  that  part  ofit...itfs  good  for 
situation  awareness. 

•  In  the  JCC,  other  than  the  Knowledge  Wall,  there  was  no  management  of  information. 

•  BW1  announces  “[SuppAreal]  update”  or  details  of  change  whenever  [SuppAreal]  has  new 
information  .... 


11  Several  critical  events  occurred  very  early  in  the  day,  making  it  difficult  to  capture  much  more  than  20  minutes  of 
activity  prior  to  the  event.  Because  information  consumers  are  not  all  immediately  aware  of  a  critical  event  and 
because  it  may  take  time  for  information  producers  to  publish  new  information,  it  was  decided  to  capture  at  least  30 
minutes  of  activity  after  an  event  to  detect  differences  in  activity  levels. 
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•  [Three  CJTF  staff]  are  discussing  information  on  K-Wall. 

•  All  staff  looking  at  K-Wall. .  .working  on  the  brief. 

K-Desk  Use 

•  The  Knowledge  Warrior  [a participant  in  the  Game]  maintained  the  K-Wall/K-Desk  and  this 
was  different,  inconsistent  across  different  cells. 

•  In  [FCC2],  we’ve  projected  two  screens  from  each  K-Desk  onto  the  large-screen  display.  In 
[FCC3],  two  screens  from  K-Desk  are  on  large-screen  display. 

•  [FCC2]  uses  K-Desk  to  get  SA  from  other  Summary  Pages,  for  the  KM  to  keep  track  of  other 
pages  and  to  update  their  own  Summary  Page  and  documents. 

•  The  commander  didn  ’t  look  at  the  K-Desk  because  he  was  always  in  chat  rooms. 

•  We  used  the  [K-Desk]  on  watch  a  lot.... 

•  In  some  cells  they  had  useful  info  on  shared  displays... others  didn’t  work  so  well. 

3.3.3.  Knowledge  Wall/Desk  Preference 

Comments  made  by  K-Wall  and  K-Desk  users  suggested  that  they  liked  the  KWVs,  particularly 
with  regard  to  the  flexibility  the  KWVs  provided  in  configuring  multiple  sources  of  information.  As 
might  be  expected,  based  on  the  difference  in  how  they  used  the  KWVs,  the  degree  of  preference  for 
the  K-Wall/K-Desk  varied  across  users  and  across  organizational  cells.  This  is  also  reflected  in  the 
difference  in  the  usage  patterns  of  the  K-Desks  when  comparing  across  the  FCCs,  as  discussed 
above.  Related  comments  were: 

•  The  K-Wall  has  been  really  good .... 

•  I  like  the  capability  to  put  any  window  where  you  want  it. 

•  I  like  the  K-Wall  (K-Web?)  approach  because  it  uses  HTML  -  shared  files. 

•  Excellent  tool! 

•  We  need  this  [touch  screen  capability]  on  the  ship  for  the  admirals  brief. 

•  I  think  the  only  thing  I  like  about  it  [the  K-Wall]  is  the  touch  screens. 

•  The  K-Wall  and  K-Desk  were  highest  ranking  tools  in  the  FCC2  survey  of  tools. 

•  ...I  liked  K-Desk  set  up,  ability  to  view  multiple  images. 
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3.4.  REVISED  KNOWLEDGE  WALL  VIEWER  HARDWARE  CONFIGURATION 


3.4.1.  Number  of  Monitors 

The  K-Wall  implemented  at  Global  2000  con¬ 
sisted  of  12  display  surfaces  (ten  21 -inch  monitors 
and  two  58.25-inch  large-screen  focus  displays), 
allowing  the  simultaneous  display  of  information 
from  multiple  sources  to  multiple  users.  However, 
both  user  comments  and  an  assessment  of  the 
visibility  of  the  information  on  the  K-Wall 
suggested  users  found  it  very  difficult  to  read  the 
information  on  the  small  monitors.  The  design  of  the  shared  K-Wall  for  Global  2001  was  changed  so 
that  information  was  presented  on  three  58.25-inch  displays.  The  K-Desks  that  were  used  in  the 
FCCs  were  composed  of  six  15-inch  displays. 

Legibility 

K-Wall  and  K-Desk  users  made  very  few  comments  about  difficulty  in  reading  the  displayed 
information.  However,  information  providers  did  occasionally  publish  Summary  Pages  with  links  to 
documents  that  were  difficult  to  read  (see  the  Business  Rules/Training  section  for  further  discussion 
of  this  problem).  The  most  commonly  observed  comments  were  about  legibility  problems  related  to 
occlusion  of  the  K-Wall  by  other  game  players.  The  following  are  examples: 

•  Can  I  move  the  Select  Monitor  window  to  another  screen?  [because  people  are  standing  in 
front  of  it], 

•  As  soon  as  we  can  get  a  clear  view  of  the  screen,  we’ll  pull  it  up. 

•  The  K-Desk  needs  to  be  big  enough  for  other  people  to  see  it. 

•  It’s  hard  to  read  that  [MS  Word  document]. 

•  It’s  hard  to  see  that  button  text  (on  Viewer)  has  changed  to  blue  from  a  distance... maybe 
change  to  red  or  some  other  color. 

•  Can  I  send  them  an  e-mail  complaining  about  their  [link]  ...It’s  hard  to  read  it. 

To  assess  the  difficulty  in  viewing  information  on  the  Overview  Page  and  on  the  Summary  Pages 
on  K-Wall,  a  “visibility  test”  using  these  formats  was  conducted.  Three  observers  with  normal  vision 
(unaided  or  corrected  to  20/20)  were  presented  an  Overview  Page  (on  Window  1  of  the  K-Wall)  and 
Summary  Pages  (on  Windows  2  and  3  of  the  K-Wall)  that  had  been  populated  with  random  text. 

Arial  font  was  used  for  all  but  one  group  of  text  characters  (where  Times  New  Roman  was  used). 
Both  font  types  were  bolded.  Several  font  sizes  (18,  20,  24,  and  38  point)  were  used  in  order  to  detect 
any  size-dependent  variations  in  visibility.  Example  stimuli  are  shown  in  Figure  25.  Viewers  were 
asked  to  identify  all  of  the  text  characters  displayed,  as  well  as  the  color  and  shape  of  any  status  indi¬ 
cators.  Accuracy  was  measured  when  the  observers  were  seated  at  each  of  four  positions  on  the  JCC 
command  table  (positions  A-D,  see  Figure  25). 
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Figure  25.  Sample  Overview  Page  (left)  and  Summary  Page  (right)  stimuli  used  in  the  visibil¬ 
ity  test.  The  top  pane  indicates  the  positions  at  which  participants  were  seated  during  the 
visibility  test. 

Identification  accuracy  was  very  high — in  most  cases  perfect.  The  lowest  average  identification 
accuracy  was  for  the  text  of  the  e-mail  address  on  the  Summary  Pages,  where  viewers  were  still  97% 
accurate  on  average.  The  mean  percent  correct  for  text  and  shape  identification  on  the  Overview 
Page  and  Summary  Pages  at  the  four  command  table  positions  are  shown  in  Table  3.  The  mean  per¬ 
cent  correct  for  identifying  each  text  and  shape  displayed  on  the  Overview  Page  and  the  Summary 
Page,  across  all  four  command  table  positions,  is  shown  in  Table  4. 


Table  3.  Mean  percent  correct  for  identification  of  characters  on  the  visibility  test  stimuli  when 
seated  at  the  four  command  table  positions. 


Position 

Distance  (inches) 

Angle  (degrees) 

Percent  Identified 

Overview 
Window  1 

Summary 
Page 
Window  2 

Summary  Page 
Window  3 

W1 

W2 

W3 

W1 

W2 

W3 

A 

115 

80  | 

67 

40 

58 

95 

99.6 

100.0 

100.0 

B 

mm 

mm 

wm 

68 

95 

100.0 

100.0 

C 

mm 

mm 

60 

EB 

98.9 

99.8 

99.8 

D 

74 

□ 

98.9 

99.9 

99.7 
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Table  4.  Mean  percent  correct  for  identification  of  characters  on  visibility  test  stimuli  for  the 
different  Overview  Page  and  Summary  Page  text  sizes. 


Overview  Page 

Text/Object 

Font 

Size 

%  Identified 

Button  Text 

Bold 
Arial, 
20  pt 

98.6 

Status  Shape 

IIS11SI 

too 

Selected  Area 

llplpt1: 

100 

Summary  Page 

Text/Object 

Font  Size 

%  Identified 

Window 

2 

Window 

3 

Title 

Bold,  Arial,  38 

Pt 

Last 

Update 

Bold,  Arial,  18 

Pt 

Status  Text 

Bold,  Arial,  24 

Pt 

100.0 

100.0 

100.0 

100.0 

Alert  Text 

Bold,  Arial,  24 
pt 

100.0 

100.0 

Alert 

Shape 

100.0 

100.0 

Alert  Color 

100.0 

100.0 

Rel.  Info 
Text 

Bold,  Arial,  24 

Pt 

100.0 

100.0 

E-Mail 

Bold,  Times  NR, 
24  pt 

97.5 

96.3 

Tradeoff  Between  Visibility  and  Amount  of  Information  Displayed 

The  tradeoff  for  increased  information  visibility  is  a  decrease  in  the  amount  of  information  that  can 
be  displayed.  Comments  related  to  the  number  of  displays  on  the  K-Wall  suggested  that  the  K-Wall 
users  thought  that  there  was  not  enough  screen  real  estate  to  display  all  the  information  that  they 
wanted  to  see.  This  problem  was  magnified  by  the  fact  that  an  entire  window  of  the  K-Wall  (Window 
1)  was  dedicated  to  the  Overview  Page  so  that,  in  effect,  K-Wall  users  only  had  two  windows  on 
which  to  display  additional  information.  Many  K-Wall,  and  some  K-Desk,  users  expressed  dissatis¬ 
faction  with  the  fact  that  the  Overview  Page  always  took  up  an  entire  monitor.  Related  comments 
included: 

K-Wall/Desk 

•  There  aren  ’t  enough  screens  to  show  everything  ( looking  at  PowerPoint  brief  from  [ FCC2 ] ) 

•  Three  screens  aren ’t  enough. .  .probably  need  four  or  five. . .  three  might  be  enough  if  you 
could  use  the  first  one.  They  want  Ops  up  here,  Intel  on  there,  they  want  the  logs.... 

•  I  liked  the  K-Desk  set  up,  ability  to  view  multiple  images. 

Overview  Page 

•  This  stuff  [  on  W1  ]  isn ’t  useful  because  it  takes  one-third  of  the  K-Wall. ...  don ’t  need  the  key 
down  in  the  comer.  Better  to  have  tabs  at  top  of  screen  showing  status  for  each  area,  that 
you  could  click  to  get  that  area.  You  could  fit  up  to  14.  You  wouldn’t  need  to  right  click  to 
move  into  another  window. 

•  I  usually  had  just  one  window  open.  ..the  Overview  screen  took  up  too  much  space. 

•  1  like  the  Overview,  but  it  only  lets  you  see  half  of  a  page. 
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3.5.  REVISED  SUMMAKER  SOFTWARE 


Most  of  the  data  related  to  the  usability  of  the  SumMaker  software  and  to  the  new/revised  features 
came  from  interviews  conducted  with  nine  information  providers  at  the  end  of  the  game.  Comments 
are  divided  into  (1)  those  relating  to  the  usability  of  the  software,  and  (2)  opinions  and  suggestions 
related  to  its  features. 

In  general,  information  providers  indicated  that  they  found  the  SumMaker  software  very  easy  to 
use  and  that  using  it  required  little  or  no  training  (10  comments).  This  was  in  spite  of  the  fact  that 
many  of  them  had  no  previous  experience  creating  Web  pages  (two  comments).  They  also  noted  that 
the  interface  of  the  software  was  simple  and  intuitive  (two  comments). 

One  usability  problem  was  associated  with  transitioning  between  the  Summary  Page  interface  and 
the  Miscellaneous  Link  interface  (two  comments).  Switching  between  the  two  interfaces  requires  the 
click  of  a  single  button;  however,  this  button  is  located  in  a  different  position  on  each  interface.  Users 
also  noted  difficulties  associated  with  moving  linked  information  (i.e.,  source  documents)  to  the 
appropriate  folder  on  the  K-Web  server.  Users  suggested  streamlining  the  steps  involved  in  publish¬ 
ing  and  saving  the  Summary  Pages  (three  comments).  A  final  problem,  that  was  in  part  an  artifact  of 
the  game,  was  the  need  to  manually  include  “game  time”  on  the  Summary  Page  every  time  it  is 
published  (three  comments).  Typically,  the  SumMaker  software  automatically  includes  the  time  from 
the  information  producer’s  computer  clock.  However,  since  the  system  time  on  the  computers  used  at 
Global  2001  did  not  correspond  to  game  time,  users  had  to  add  game  time  manually.  SumMaker 
users  expressed  frustration  in  needing  to  do  this. 

The  information  providers  indicated  that  they  believed  SumMaker  to  be  a  good  and  useful  program 
that  met  their  needs  (five  comments).  They  liked  the  template  approach,  which  standardized  their 
input  (three  comments),  although  one  user  indicated  that  he  was  dissatisfied  with  the  lack  of 
flexibility  in  changing  the  layout  of  the  Summary  Page.  They  also  liked  the  ability  to  use  MS  Office 
products  and  to  link  to  products  already  created  for  other  purposes  (four  comments).  Information 
providers  also  expressed  a  preference  for  the  ability  to  drag  and  drop  information  into  the  fields  of 
the  interface  (two  comments). 

Several  features  were  suggested  by  the  information  providers  that  may  reflect  new  user  require¬ 
ments.  Many  of  them  indicated  a  need  to  archive  old  information  (Summary  Pages,  linked  docu¬ 
ments)  (five  comments).  One  user  suggested  that  a  phone  book  be  provided  with  the  software  and 
another  suggested  that  SumMaker  be  linked  to  collaboration  systems.  (See  also  discussion  of  Feed¬ 
back  to  Information  Providers,  next  section).  Comments  about  the  usability  and  features  of  the  Sum- 
Maker  software  included: 

SumMaker  Usability 

•  It  made  it  easy  to  produce  Web  pages  with  minimal  training. 

•  I  wish  [other  tools]  were  as  obvious.. .wish  it  were  all  this  easy. 

•  ...I  never  published  a  Web  page  before  I  got  here. 

•  It  had  a  clean  Interface. ..simple  and  intuitive. 

•  The  transition  between  SumMaker  and  Miscellaneous  Links  interface  was  not  obvious. 

•  It  was  confusing  that  you  had  to  copy  files  into  a  separate  directory — this  step  should  be 
taken  away  from  the  user. 


37 


•  It  would  have  been  nice  if  the  computer  clock  had  been  set  to  game  time.  It  wasn  ’t  that  easy 
to  add  the  time. ...Nice  feature  to  have  the  time  stamp  though. 

SumMaker  Features  and  New  Requirements 

•  It  was  a  useful  way  to  present  information  from  [SuppArea2]. 

•  Nice  to  have  a  template,  to  be  able  to  keep  updating.... 

•  It  supported  MS  Office  products  and  file  formats. 

•  There  was  a  loss  of  flexibility  (because  of  the  same  reasons  that  make  it  sailor-proof). 

•  I  liked  the  fact  that  you  could  drag-and-drop  information  into  SumMaker. 

•  Users  need  the  ability  to  archive  old  information. 

•  A  phone  book  feature  on  Summary  Pages  would  be  nice  (mouse  over  for  details  about  per¬ 
son,  hyperlinks  to  each ). 

•  You  might  want  a  link  to  another  collaboration  system  (e.g.,  whiteboard). 


3.5.1.  Increased  Link  capacity 

The  SumMaker  software  allowed  information  pro¬ 
ducers  to  easily  link  products  to  the  Summary  Pages 
they  created.  The  revised  SumMaker  software 
allowed  for  up  15  links  (i.e.,  one  in  each  of  the  three 
status  areas,  a  graphic,  seven  Alerts  and  Impacts,  and 
four  Related  Information  links).  Information  produc¬ 
ers  were  encouraged  to  link  to  “existing”  information 
(e.g.,  products  found  in  WIGS  and  products  that  had 
been  created  for  other  purposes  by  the  information  producer)  and  to  cross-reference  information  from 
other  organizational  cells  in  the  K-Web.  Figure  26  shows  the  average  number  of  links  that  were 
included  on  Summary  Pages.  On  average,  information  producers  included  8.9  links  (5.8,  10.9,  and 
10.0  links  per  page  for  Phases  1,  2,  and  3,  respectively). 
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Figure  26.  The  number  of  links  included  on  the  Summary  Pages. 
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Figure  27.  The  format  of  information  linked  to  the  Summary  Pages. 

Figure  27  shows  the  format  of  the  information  linked  to  the  Summary  Page.  Links  were  most  com¬ 
monly  in  HTML  format,  although  information  producers  frequently  included  MS  PowerPoint  docu¬ 
ments  and  graphics  as  links. 

Recorded  comments  also  suggest  that  many  information  producers  took  advantage  of  the  informa¬ 
tion  posted  by  other  information  providers  to  the  K-Web,  to  update  their  own  Summary  Pages  and 
other  products.  These  comments  included: 

•  The  BWC  is  telling  others  how  their  information  will  be  a  link  from  the  CJ  IF  Summary 
Page. 

•  [FCC1]  indicates  that  their  information  for  evening  brief  is  in  a  link  from  their  Summary 
Page. 

•  [FCC2]  uses  K-Desk  to  get  SAfrom  other  Summary  Pages,  for  the  KM  to  keep  track  of  other 
pages  and  to  update  their  own  Summary  Page  and  documents. 

•  The  CINC  [ Summary  Page ]  has  a  link  to  ROEs,  can  you  put  it  on  our  Summary  Page ? 

•  The  CINC  page  has  a  link  to  [SuppAreal]  links. 


3.5.2.  Feedback  to  Information  Providers 


Comments  made  by  information  providers  using 
the  previous  version  of  the  SumMaker  software 
(during  Global  2000)  indicated  that  they  wanted 
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from  whom  they  could  receive  feedback.  When 

SumMaker  users  published  a  new  Summary  Page,  the  time  that  it  was  posted  to  the  K-Web  was  indi¬ 
cated  on  the  SumMaker  interface.  When  the  designated  primary  watchstander  accessed  the  Summary 
Page  (using  the  KWV),  the  time  that  the  page  was  viewed  was  automatically  reflected  on  the  inter¬ 
face.  Information  providers  were  thus  given  information  about  when  the  K-Wall  users  had  looked  at 
their  page  and  could  compare  that  to  the  time  at  which  they  published  it.  (Note  that  this  type  of 
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feedback  supports  the  change  alerting  functionality  provided  by  the  Summary  Pages,  allowing  the 
SumMaker  users  to  post  change  diamonds  based  on  knowledge  of  whether  and  when  K-Wall  users 
had  viewed  their  Summary  Page.) 

Comments  made  by  information  providers  (and  other  game  players)  indicated  that  many  of  them 
did  not  believe  they  were  receiving  adequate  feedback  about  the  use  of  their  Summary  Pages.  Some 
of  these  comments  suggest  that  SumMaker  users  were  not  aware  of  the  automatic  feedback  provided 
by  the  software.  SumMaker  and  Summary  Page  user  comments  were: 

Feedback  related  to  Summary  Page  access 

•  You  need  to  have  feedback  that  people  are  using  your  information. 

•  1  don ’t  know  what  the  CINC  is  getting  from  us  [  the  CJTF  staff] — I  know  it’s  on  our  page  but 
I  don ’t  know  that  the  CINC  is  looking  at  it. 

•  You  need  information  about  who ’s  viewed  your  information  ( specifically  who)  so  you  can 
talk  to  the  person  about  it — even  if  in  the  form  of  e-mail. 

3.6.  REVISED  KNOWLEDGE  WEB  VIEWER  SOFTWARE  DESIGN 


3.6.1.  Revised  Overview  Page 

The  revised  KWV  software  implemented  at  Global 
2001  featured  a  dedicated  Overview  Page  on  Win¬ 
dow  1 ,  which  facilitated  attention  management  and 
information  integration.  The  revised  Overview  Page 
design  is  described  in  Section  1.2.4  above.  Specific 
features  of  this  page  include: 

•  Easy  access  to  Summary  Pages. 

•  Overall  status  of  all  areas  over  time. 

•  Indications  of  new  or  changed  status  information. 

Data  related  to  each  of  these  features  are  discussed  separately. 

Easy  access  to  Summary  Pages.  Each  button  on  the  Overview  Page  allowed  users  to  quickly  and 
easily  display  information  about  an  operational  cell.  When  users  selected  a  button  with  their  mouse,  a 
portion  of  the  selected  Summary  Page  was  displayed  on  the  right  side  of  the  Overview  Page.  If  users 
wanted  to  drill  down  for  further  information,  they  could  display  the  entire  Summary  Page  on  one  of 
the  other  KWV  windows  by  right-clicking  on  the  button.  This  brought  up  a  “select  Window”  control, 
which  allowed  them  to  move  an  entire  Summary  Page  to  the  window  of  their  choice  within  the  KWV 
software. 

K-Wall  and  K-Desk  users  frequently  right-clicked  on  buttons  to  move  Summary  Pages  from  the 
Overview  Page  window  to  other  windows  of  the  K-Wall  or  K-Desk.  Figure  28  indicates  that  the 
CJTF  K-Wall  users,  especially,  used  the  left  and  right  mouse  buttons  to  take  advantage  of  this  capa¬ 
bility.  (Other  means  of  moving/manipulating  information  using  the  KWV  software  will  be  discussed 
in  Section  3.6.2  Revised  KWV  Software:  Usability.) 
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Figure  28.  KWV  operations  on  the  Overview  Page  to  display  Summary 
Pages  (left  mouse  click)  and  move  them  to  windows  on  the  K-Wall  and 
K-Desks  (right  mouse  click). 

Overall  Status  of  all  Areas  Over  Time  and  New  or  Changed  Information 

Using  colored  rectangles  (for  existing/known 
information)  and  diamonds  (for  new  or  changed 
information),  the  status  bar  on  the  Overview  Page 
shows  the  predicted  status  of  each  functional  area 
over  three  time  periods:  Today  (the  next  24  hours), 

Tomorrow  (24  to  48  hours),  Long-Range  (beyond 
48  hours.  This  design  allows  KWV  users  to  easily 
integrate  information  from  these  areas  “at  a  glance,” 
allowing  rapid  appraisal  of  (1)  the  predicted  status  over  time  for  all  of  the  operational  cells  and,  (2) 
whether  the  reported  status  information  represents  new  or  changed  information.  Further,  the  color  of 
text  on  the  buttons  of  the  Overview  Page  changed  from  black  to  blue  whenever  the  Summary  Page 
corresponding  to  that  button  had  been  updated.  (This  change  only  occurred  on  a  given  KWV  for 
those  operational  areas  that  had  selected  that  KWV  as  their  “primary  watchstander.”  During  Global 
2001,  all  of  the  FCCs  and  the  supplemental  areas  connected  to  the  CJTF  K-Wall  as  their  primary 
watchstander.  The  CJTF  information  provider  connected  to  the  CINC  K-Desk.) 

Surprisingly,  no  comments  were  made  related  to  the  status  indicators.  However,  some  CJTF  K- 
Wall  users  did  periodically  discuss  the  status  of  operational  cells  in  terms  of,  for  example,  “things 
turning  yellow”  (i.e.,  from  green,  indicating  a  deterioration  in  status).  This  suggests  that  these  users 
were  gaining  status  information  from  the  Overview  Page. 

One  indication  that  KWV  users  took  advantage  of  integrated  status  information  would  be  whether 
they  were  more  likely  to  access  Summary  Pages  from  operational  cells  that  indicated  that  their  pre¬ 
dicted  status  was  degraded  (i.e.,  a  change  from  green  to  yellow,  green  to  red,  or  yellow  to  red). 
Analysis  of  the  CJTF  KWV  log  (which  indicates  access  times)  and  the  automatically  captured  Sum- 
Maker  data  files  (which  indicate  update  times)  indicated  that  this  was  the  case,  at  least  for  the  CJTF 
K-Wall.  As  can  be  seen  in  Figure  29,  Summary  Pages  that  indicated  deterioration  in  status  were 
reflected  on  the  Overview  Page  and  were  accessed  more  quickly  overall  (17  minutes  after  they  were 
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published)  than  those  that  indicated  improved  or  unchanged  status  (36  and  35  minutes  after  being 
published,  respectively). 


* 


|  SPs  Showing 


Degraded  Status 


SPs  Showing 
Improved  Status 


■  Phase  1 1 
®  Phase  2 
□  Phase  3 1 
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Figure  29.  Access  time  (by  the  CJTF  KWV)  for  Summary  Pages  as  a  func¬ 
tion  of  type  of  change  in  status  over  time. 

It  should  be  noted  that,  because  very  few  yellow  and  red  status  indicators  were  posted,  these  results 
are  based  on  a  very  small  amount  of  data  and  should  be  interpreted  with  caution.  (A  total  of  only 
seven  Summary  Pages  indicated  degraded  status  and  five  Summary  Pages  indicated  improved  status 
across  all  operational  cells.) 

Figure  30  shows  the  average  number  of  minutes  that  it  took  CJTF  K-Wall  users  to  access  a  Sum¬ 
mary  Page  after  it  had  been  published.  These  data  only  include  the  access  times  for  published  Sum¬ 
mary  Pages  that  produced  visual  change  in  the  Overview  Page  when  they  were  posted.  Pages  that 
caused  no  change  in  the  status  bar  when  published  were  excluded  from  the  analysis.12  On  average, 
Summary  Pages  were  accessed  30  minutes  after  they  were  published,  though  access  times  generally 
decreased  during  the  course  of  the  game,  suggesting  that  users  became  more  familiar  with  the  use 
and  intent  of  change  diamonds  as  the  game  progressed. 


12  For  example,  an  information  provider  may  have  published  a  Summary  Page  before  the  CJTF  K-Wall  users  had 
accessed  the  last  page  that  they  posted.  If  this  new  page  did  not  include  any  new  information  about  status,  there 
would  be  no  indication  of  it  on  the  K-Wall  (the  button  would  already  be  blue  and  the  status  representation  would 
remain  the  same). 
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Figure  30.  Access  time  (by  the  CJTF  KWV)  for  Summary  Pages  as  a  func¬ 
tion  of  the  presence  of  change  diamond. 

Figure  30  also  shows  the  difference  in  access  time,  relative  to  when  information  providers  pub¬ 
lished  a  Summary  Page,  as  a  function  of  whether  or  not  the  Summary  Page  contained  a  change  dia¬ 
mond.  Interestingly,  Summary  Pages  with  change  diamonds  were  accessed  less  quickly  (47  minutes 
after  being  published)  than  those  without  change  diamonds  (31  minutes  after  being  published).  This 
surprising  fact  is  difficult  to  interpret;  however,  it  has  been  suggested  that  perhaps  users  felt  more 
comfortable  with  areas  that  they  knew  were  being  well  managed  (as  evidenced  by  change  indicators) 
and  therefore  did  not  feel  the  need  to  access  these  areas  as  quickly  or  as  often  as  those  that  showed 
no  activity. 

Information  providers  also  made  several  suggestions  related  to  the  current  KWV  and  K-Web 
change  alerting  functionality,  indicating  a  need  for  more  effective  change  alerting.  Many  of  these 
comments  reflect  the  fact  that  K-Wall  users  did  not  find  the  change  in  button  text  from  black  to  blue 
to  be  a  salient  indicator  of  recently  published  information.  They  also  expressed  dissatisfaction  with 
the  inconsistent  use  of  change  diamonds  by  information  providers  on  the  Summary  Pages.  In  par¬ 
ticular,  they  indicated  that  more  information  about  changes  should  be  indicated  on  the  K-Wall  (such 
as  when  change  diamonds  were  included  in  the  Alerts  and  Impacts  section  of  a  Summary  Page)  or  on 
the  Summary  Pages  themselves  (such  as  when  new  information  is  available  on  a  link  from  that  page). 
Sample  comments  were: 

•  Things  are  turning  blue,  telling  me  that  there  are  changes  so  I’m  going  to  check  them. 

•  [SuppAreal  ]  IP  verbally  notifies  BW1  and  shows  him  his  page  It’s  faster  than  blue  text  on 
viewer. 

•  They  have  blue  stuff  (new  hyper  links  on  the  Summary  Page)  but  no  change  diamonds. 

•  You  need  something  that  pulses  for  ~20  seconds  when  something  changes  to  a  diamond 
...need  to  know  that  information  has  changed.  Blue  text  is  not  an  “alert” — need  something 
more  salient. 

•  I  wish  they  had  status  for  Alerts  &  Impacts  on  K-Wall.. .we  changed  Today,  Tomorrow,  LR 
status  once  a  day  but  changed  Alerts  more  often. 
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•  It  should  indicate  on  Summary  Page  when  you’ve  updated  a  linked  document,  e.g.,  change 
the  Today  link  to  a  diamond. 

•  It  was  easy  to  just  make  changes  to  files... without  having  to  republish  everything. 

•  You  can  post  things  on  the  Web  all  day  long  but  you  need  to  push  information... can’t  assume 
people  will  pull  off  Web. 

General  comments  related  to  the  Summary  Page  included: 

•  ...it’s  easy  to  use. 

•  The  Overview  Page/K-Web  could  almost  do  away  with  voice  communications. 

•  The  WIGS  Web  page  (an  overview  page  on  WIGS)  was  not  intuitive  because  it  didn  ’t  match 
the  Overview  Page  on  the  KWV. 

3.6.2.  Revised  Knowledge  Web  Viewer  Software:  Usability 

The  revised  KWV  software  provided  several  functions  that  allowed  users  to  configure  KWV  con¬ 
tent,  update  information,  and  run  other  software  tools.  The  Overview  Page  allowed  users  to  easily 
bring  up  Summary  Pages  and  move  them  to  other  windows  (see  section  3.6.1  Revised  Overview 
Page).  The  menu  bar  on  each  window  included  controls  common  to  Web  browsers  (see  Figure  31).  A 
control  menu  (see  Figure  32),  accessible  from  each  window,  allowed  users  to  hide  or  show  underly¬ 
ing  applications  (e.g.,  MS  PowerPoint,  tactical  displays,  SumMaker,)  and  gave  users  access  to  two 
preset  configurations  (Preset  1 ,  in  which  only  Windows  1  through  3  were  shown,  and  a  default 
setting,  in  which  all  six  windows  were  shown). 

<®|«  |  0  i  ’£■  i  iL'  I S559HS2!  -j  Go|  Siowm  |  .  ■  W-  H;  Uontiol  I  Clote  All  j 

Figure  31 .  The  KWV  software  menu  bar. 
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Figure  32.  KWV  control  menu. 

Analysis  of  the  KWV  data  logs  indicated  that  users  took  advantage  of  these  capabilities  differently 
in  each  of  the  operational  cells.  Figure  33,  Figure  34,  and  Figure  35  show  how  often  operators  of  the 
CJTF  K-Wall,  the  CINC  K-Desk  and  FCC  K-Desks,  respectively,  used  the  various  KWV  functions. 
CJTF  K-Wall  users  frequently  reconfigured  the  display  using  the  left  and  right  mouse  click  opera¬ 
tions  on  the  Overview  Page.  Users  of  the  CINC  K-Desk  were  more  likely  to  move  information  into 
KWV  windows  via  the  Channels  menu  (a  pull-down  list  that  provided  access  to  the  operational  cell 
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Summary  Pages,  as  well  as  to  WIGS),  and  they  regularly  used  the  Refresh  command.  The  FCC 
K-Desk  operators  were  equally  likely  to  access  information  using  these  two  methods.  They  also  fre¬ 
quently  used  the  Hide  command  (via  the  Control  Menu),  probably  in  order  to  display  tactical  infor¬ 
mation  and/or  the  SumMaker  software  on  the  K-Desks  (see  Figure  35).  Overall,  though,  the  FCC  K- 
Desk  operators  used  the  various  KWV  operations  much  less  often  than  either  the  CJTF  K-Wall  users 
or  the  CINC  K-Desk  users. 


Figure  33.  KWV  software  use  on  the  CJTF  K-Wall. 


Figure  34.  KWV  software  use  on  the  CINC  K-Desk. 
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Figure  35.  KWV software  use  on  the  FCC  K-Desk. 

Most  KWV  operator  comments  suggested  they  liked  the  software,  especially  the  ease  with  which  it 
allowed  them  to  move  information  around.  One  user  of  the  K-Wall  expressed  dissatisfaction  with  the 
method  of  moving  information  into  another  window  via  right-clicking  a  button  on  the  Overview 
Page.  One  problem  expressed  by  many  users  was  the  inability  to  predict  in  which  window  of  the 
KWV  dialogue  boxes  would  appear  (i.e.,  not  necessarily  the  “expected”  window,  or  the  one  in  which 
the  user  was  working).  This  problem  was  not  caused  by  the  KWV  software  per  se,  but  was  due  to  the 
way  that  MS  Windows  handles  multiple  monitors.  The  comments  related  to  KWV  usability  issues 
were: 

•  It  was  easy  to  display  information  on  the  K-Wall. 

•  The  K-Wall  helped  us. . .BWC  was  comfortable  using  it. ... 

•  I  wish  there  was  an  easier  way  to  put  stuff  on  windows  other  than  right-clicking. 

•  Problem  that  you  didn ’t  know  where  a  window  would  pop  up. 

•  Some  confusion  arose  on  the  K-Wall  when  more  than  one  person  attempted  to  operate  the 
KWV  simultaneously  (K-Wall  KWV  control  could  be  implemented  via  two  sets  of  mouse 
and  keyboard  controls  or  the  touch  screens). 

•  Problems  when  we  wanted  to  run  a  PowerPoint  slide  show. ..when  we  clicked  on  another 
window,  slides  advanced  (because  it’s  a  single  desktop)... you  should  be  able  to  keep  func¬ 
tions  running  in  the  background. 


3.7.  OTHER  EMERGING  THEMES 
3.7.1.  Training  and  Business  Rules 

The  training  documents  provided  to  K-Web  tool  users  were  described  in  the  Training  and  Business 
Process  section  of  the  Introduction  (Section  1.3.3).  Information  providers  and  users  of  the  KWVs 
were  given  individual  training  on  the  K-Web  tools  they  would  be  using  during  the  game.  Training 
was  conducted  by  K-Web  designers  prior  to,  or  during,  the  game  as  the  schedule  and  events  allowed. 
Training  typically  involved  reviewing  the  K-Web  business  rules  brief  and  appropriate  operator 
instructions.  These  presentations  were  immediately  followed  by  one-on-one  tutoring  in  the  use  of 
K-Web  tools.  Unfortunately,  training  was  sometimes  not  given  until  the  second  or  third  day  of  the 
game  because  some  users’  roles  were  not  identified  until  then.  Further,  K-Web  training  was  not  given 
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to  all  game  players,  even  though  they  would  potentially  use  the  K-Web  products  (i.e.,  Summary 
Pages  and  their  links).  Rather,  K-Web  training  was  only  given  to  identified  information  providers 
and  KWV  operators. 

Users  who  did  have  contact  with  K-Web  training  materials  believed  them  to  be  very  useful.  How¬ 
ever,  much  of  the  data  in  the  results  section  points  to  a  lack  of  understanding  of  (or  lack  of  adherence 
to)  the  K-Web  business  rules.  Specifically,  this  includes,  but  is  not  limited  to: 

•  insufficient  knowledge  about  the  availability  of  feedback  between  information  providers  and 
consumers; 

•  inconsistent  use/overuse/under  use  of  content  features  (e.g.,  the  status  indicator  shapes  and 
colors)  on  the  Summary  Pages; 

•  indiscriminate  access  of  information  products  by  consumers;  and 

•  variability  in  use  of  the  K-Web  tools  across  the  operational  cells. 

Many  other  observations  and  recorded  comments  further  suggest  a  lack  of  understanding,  by  many 
players,  of  the  K-Web  concept  and  business  rules.  The  following  are  examples: 

Training 

•  The  training  documents  were  fairly  intuitive. .  .didn  ’t  have  problems  figuring  things  out. 

•  It  would  have  been  nice  if  the  WIGS  training  had  let  people  know  the  Summary  Pages  were 
there. 

K-Web  Concept 

•  How  do  we  know  that  the  Component  Commanders  know  what’s  going  on? 

•  Does  the  CINC  have  access  to  the  same  information  as  we  do? 

•  We  need  help/advice  about  how  to  take  advantage  of  the  wall. 

•  The  system  assumed  people  knew  where  to  go  for  information. 

•  There  should  have  been  business  rules  about  how  to  share  information. . .. 

•  Fusing,  focusing,  displaying  information  is  different  at  different  levels. 

Specific  K-Web  Business  Rules 

•  We ’ve  fought  the  war  today  exactly  the  way  we  weren ’t  supposed  to. ...with  PowerPoint. 

•  They  used  diamonds  in  Alerts  and  Impacts  but  not  in  the  Today,  Tomorrow,  Long-Range 
area. 

•  There  has  been  a  lot  of  debate  regarding  content:  Where  [on  information  products]  should 
information  be  red — [i.e,.  if  not  in  status  over  time]. 

•  I  didn’t  feel  like  ROE  for  use  of  the  system  was  established,  e.g.,  when  to  use  a  green  dia¬ 
mond  vs.  a  square,  how  long  a  diamond  should  be  up.... 

•  The  Admiral  did  not  know  what  the  colors  meant....  You  need  some  sort  of  key  to  the 
symbology. . .. 
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4.  GENERAL  DISCUSSION 


Based  on  the  K-Web  use  evaluation  from  Global  2000,  several  K-Web  tool  design  changes  were 
implemented  at  Global  2001.  The  primary  focus  of  this  discussion  is  the  evaluation  of  the  revised 
K-Web  design  features  in  relation  to  the  user  needs  identified  during  the  Global  2000  games.  Specifi¬ 
cally,  data  analyses  of  observational  and  automatically  collected  data  lent  to  the  assessment  of 
whether: 

•  The  K-Web  information  products,  and  the  KWVs  and  the  support  tools,  supported  other 
previously  identified  user  requirements. 

•  The  revised  features  of  the  K-Web  information  products,  and  the  KWVs  and  K-Web  support 
tools,  met  the  information  integration,  cognitive  support,  and  collaboration  needs  identified 
by  the  previous  Global  2000  use  evaluation. 

•  There  were  user  needs  not  being  met  by  the  current  K-Web  products  and  tools. 

•  There  were  ergonomic  problems  related  to  the  K-Web  products  and  tools. 

•  Users  preferred  the  K-Web  products  and  tools  to  other  information  tools. 

The  following  discussion  summarizes  the  results  described  above  and  when  possible,  within  the 
context  of  a  comparison  to  the  Global  2000  findings. 

4.1.  SUMMARY  PAGES 

Summary  Pages  were  continuously  updated  and  accessed  throughout  each  phase  of  the  Global 
2001  war  game.  As  a  result.  Summary  Pages  and  their  links  composed  K-Desks  and  K-Wall  content 
the  majority  of  time  during  Global  2001.  Data  were  not  collected  by  the  SumMaker  software  during 
the  Global  2000  game;  therefore,  the  content  of  the  Summary  Pages  and  their  links  cannot  be  com¬ 
pared  across  Global  2000  and  2001.  Observational  data,  however,  do  allow  some  comparison  of  the 
use  of  and  preference  for  Summary  Pages.  Comments  and  observations  made  during  both  war  games 
suggest  that  the  players  found  the  Summary  Pages  to  be  very  useful  for  tracking  the  operational 
situation  and  staying  apprised  of  other  players’/cells’  activities  and  concerns.  Overall,  the  revised 
standardized  Summary  Page  format  was  considered  easy  to  use  and  successful  in  terms  of  meeting 
the  needs  of  both  information  providers  and  consumers. 

During  Global  2000,  information  providers  and  consumers  stated  a  need  for  a  means  of  alerting 
other  users  to  important  changes  in  Summary  Pages  and  their  links.  The  revised  Summary  Page 
design  developed  for  the  Global  2001  game  addressed  this  need  with  the  change  alerting  functional¬ 
ity  (change  diamonds)  on  the  Summary  Pages  and  the  KWV  Overview  Page.  Information  providers 
took  advantage  of  this  functionality  during  Global  2001  by  including  change  diamonds  in  a  high  pro¬ 
portion  of  Summary  Pages.  The  excessive  use  of  change  alerts,  however,  could  be  problematic;  when 
change  alerts  are  always  present,  users  tend  to  ignore  them  (Mayhew,  1992).  The  use  of  change  dia¬ 
monds  only  when  there  has  been  an  actual  change  in  status  should  yield  even  stronger  support  for 
this  tool. 

Improved  training  regarding  business  rules  should  lead  to  more  appropriate  use  of  change  dia¬ 
monds  in  Summary  Pages.  Attention  research  literature  points  to  the  fact  that  it  is  very  difficult  for 
people  to  keep  track  of  changes,  especially  when  they  are  interrupted  or  switching  between  tasks 
(e.g.,  Rensink,  O’Regan,  &  Clark,  2000).  Changes  in  a  busy,  multi-task  environment,  such  as  a 
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military  command  center,  must  be  salient  and  meaningful  enough  to  be  detected.  Adequate  training 
is  necessary  to  provide  users  with  a  relevant  “attentional  set”  for  the  indicators  of  important  change 
(Folk,  Remington,  &  Johnston,  1992).  Other  attention  management  and  change  detection  issues  that 
need  to  be  examined  in  the  context  of  K-Web  information  products  include: 

•  How  to  represent  when  the  change  indicator  was  added  to  the  Summary  Page  (in  real  time 
and/or  relative  to  other  posted  changes). 

•  How  to  determine  which  Summary  Page  users  will  find  changes/alerts  regarding  changes 
most  relevant  (to  allow  tailoring  of  change  alerting  to  users). 

•  How  to  automate  change  alerting. 

The  Global  2001  Summary  Page  design  included  the  information  producer’s  e-mail  address  so  that 
users  viewing  the  Summary  Page  could  send  feedback  regarding  page  content.  Many  information 
consumers  were  observed  providing  feedback  to  the  producers  in  the  form  of  e-mail  or  other  com¬ 
munications.  However,  information  producers  expressed  dissatisfaction  in  the  amount  of  feedback 
that  they  received  about  the  content  of  their  information  products,  suggesting  that  a  better  mechanism 
or  tool  for  providing  content-related  feedback  should  be  included  in  future  Summary  Page  designs. 

4.2.  KNOWLEDGE  WALL/DESKS 


4.2.1.  Knowledge  Wall/Desks  Use 

Observations  made  during  the  Global  2001  and  2000  games  (see  Appendix  B)  pointed  to  differ¬ 
ences  in  K-Wall  use  by  the  different  CJTF  staffs.  One  of  the  most  significant  differences  was  the 
increased  role  that  the  K-Wall  played  as  a  briefing  tool  at  Global  2001. 

A  noteworthy  finding  from  the  Global  2000  evaluation  was  the  impact  that  the  K-Wall  had  on 
eliminating  the  need  for  a  traditional  8-hour  briefing  cycle.  That  K-Wall  was  used  as  a  means  of 
visualizing  mission-status  in  a  continuous  fashion,  through  the  display  of  K-Web  Summary  Pages 
and  other  information  products.  Its  users  required  only  occasional,  impromptu  briefs,  which  occurred 
as  the  operational  situation  demanded;  these  briefs  were  typically  conducted  via  the  display  of  exist¬ 
ing  K-Web  products.  Users  emphasized  how  the  radical  change  in  briefing  process  and  purpose 
effectively  changed  and  improved  their  overall  Concept  of  Operations. 

The  Global  2001  K-Wall,  in  contrast,  was  used  less  as  an  SA  tool  than  the  previous  year,  and  more 
as  a  briefing  tool.  The  K-Wall  was  used  to  brief  the  CJTF  staff  daily  and  to  prepare  the  CJTF’s  daily 
brief  to  the  CINC.  Most  content  for  the  briefs  consisted  of  PowerPoint  slides  created  expressly  for 
this  purpose. 

K-Wall  data  collected  at  each  game  also  facilitates  a  comparison  of  K-Wall  use  across  the  two  war 
games.  Figure  13,  which  shows  the  number  of  Summary  Pages  accessed  on  the  KWVs,  indicates  that 
users  of  the  Global  2000  K-Wall  accessed  almost  3.5  times  as  many  Summary  Pages  using  the  KWV 
software.  This  result  is  to  be  expected  since  there  were  12  displays  on  the  Global  2000  K-Wall  on 
which  to  display  information  (vice  three  on  the  2001  K-Wall).  Conversely,  the  number  of  links 
accessed  via  the  KWV  software  in  2001  was  almost  double  that  of  2000.  This  too  is  probably  attrib¬ 
utable  to  the  number  of  display  surfaces  available. 

Screen  captures  of  the  two  K-Walls  (the  2000  vs.  the  2001  K-Walls)  also  indicate  a  difference 
in  the  type/format  of  information  displayed  on  them.  Figure  36  shows  the  percentage  of  time  that 
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K-Web  products  and  other  information  products  were  displayed  on  the  Focus  Monitors  (two  large 
central  monitors)  of  the  2000  K-Wall,  all  12  monitors  of  the  2000  K-Wall,  and  Windows  2  and  3  on 
the  2001  K-Wall.  In  addition  to  the  differences  in  the  usage  patterns  of  K-Web  products,  there  were 
also  differences  in  the  amount  of  time  that  tactical  displays  and  other  software  were  displayed.  Users 
during  Global  2001  never  displayed  tactical  information  on  the  K-Wall,  whereas  tactical  information 
was  a  common  focus  during  the  previous  year.  The  reason  for  this  is  most  likely  the  fact  that  the  tac¬ 
tical  software  used  during  Global  200113  was  run  on  other  shared  displays.  The  screen  captures  also 
indicate  that  the  Global  2001  users  were  more  likely  to  run  the  PowerPoint  software  on  the  K-Wall, 
supporting  the  observational  finding  that  it  was  used  frequently  to  prepare  and  display  briefs.  Finally, 
the  Global  2001  K-Wall  users  were  more  likely  to  display  WIGS  products  on  the  K-Wall.  In  particu¬ 
lar,  the  CJTF  watch  log  was  frequently  displayed  and  updated  on  Window  3  of  the  K-Wall. 
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Figure  36.  Format  of  information  displayed  on  the  Global  2000  and  2001  K-Walls. 

As  would  be  expected,  the  K-Desks  were  used  differently  than  the  K-Walls,  and  use  of  the 
K-Desks  varied  across  operational  cells.  One  of  the  intents  of  K-Desk  design  was  to  support  both 
information  production  and  information  consumption.  The  K-Desks  were  used  in  both  ways  during 
Global  2001;  they  were  used  for  the  maintenance  of  SA,  as  well  as  for  the  creation  and  revision  of 
information  products  and  for  collaboration  with  others. 

4.2.2.  Knowledge  Wall/Desks  Legibility 

The  Global  2000  K-Wall  evaluation  found  that  users  had  difficulty  reading  the  information  on 
small  peripheral  monitors  of  the  K-Wall  implemented  for  that  game.  In  response  to  this  problem,  the 
design  of  the  shared  K-Wall  at  Global  2001  was  changed  so  that  it  consisted  of  three  large  monitors. 
Visibility  assessments  taken  during  both  games  indicated  that  identification  accuracy  of  information 
on  the  K-Wall  improved  from  an  average  of  ~91%  in  2000  to  near  100%  in  2001.  However,  this 


13  BattleScape  NT  (http://www.autometric.com/NEW/Products/Visualization/bs.html)  and  EDGE  Viewer 
(http://www.autometric.com/NEW/Products/Visualization/viewer.html) 
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came  at  the  cost  of  a  reduction  in  the  amount  of  information  being  displayed — a  trade-off  that  many 
users  found  unsatisfactory. 

4.2.3.  Information  Age  and  Source 

Global  2000  users  also  expressed  a  need  for  information  about  the  age  and  source  of  information 
products  in  the  K-Web.  This  was  not  addressed  in  the  2001  Summary  Page  or  SumMaker  designs. 
Future  K-Web  design  efforts  will  attempt  to  meet  these  needs. 

4.3.  THE  SUMMAKER  SOFTWARE 

The  SumMaker  tool  proved  to  be  successful  at  meeting  the  needs  of  information  providers  during 
both  Global  2000  and  2001  war  games.  Users  who  had  no  experience  publishing  Web  pages  were 
able  to  do  so  with  little  or  no  training.  Of  course,  suggestions  for  improvements  to  the  SumMaker 
software  tool  were  made  during  both  games.  In  addition  to  populating  the  basic  information  fields  of 
the  Summary  Page  format,  SumMaker  users  exercised  their  option  to  link  additional  information 
sources  to  their  pages;  Summary  Page  viewers  accessed  these  frequently.  In  fact,  user  comments 
indicated  the  need  to  add  more  links  to  Summary  Pages.  With  so  much  information  being  linked  to 
Summary  Pages,  the  need  has  developed  for  better  link  organization  and  navigation.  User  comments 
also  indicated  that  the  standardized  format  available  through  SumMaker  made  it  easier  for  informa¬ 
tion  providers  to  create  and  update  their  pages. 

Information  providers  at  Global  2000  expressed  a  need  for  feedback  from  information  consumers 
about  use  of  their  products.  Although,  the  revised  SumMaker  software  provided  two  methods  of 
feedback  (in  the  forms  of  an  e-mail  link,  and  a  visual  indication  regarding  whether  the  watchstander 
had  viewed  the  Summary  Page),  comments  made  by  Global  2001  information  providers  indicated  a 
desire  for  additional  feedback  regarding  the  use  of  their  products.  Some  of  these  comments  suggest 
that  SumMaker  users  were  not  aware  of  the  automatic  feedback  provided  by  the  software  that  indi¬ 
cated  whether  the  watchstander  had  viewed  the  Summary  Page.  Even  for  those  who  did  view  such 
feedback,  it  was  judged  to  be  insufficient  in  communicating  information  the  consumer  needs.  This 
was  evidenced  by  comments  from  consumers,  such  as  senior  staff,  indicating  that  they  were  not 
getting  the  information  they  needed  when  they  needed  it,  and  from  information  producers  who  could 
not  determine  whether  the  information  they  were  providing  was  useful  to  the  information  consumers. 
These  results  are  due,  in  part,  to  the  insufficient  training  of  both  the  information  providers  and  infor¬ 
mation  consumers  regarding  appropriate  Summary  Page  content  and  the  available  feedback  tools. 
However,  it  also  suggests  a  need  for  a  different  mechanism  for  providing  feedback.  Timely  and  rele¬ 
vant  feedback  can  be  an  important  motivating  factor  in  ensuring  successful  task  performance  (Pinder, 
1998).  In  the  case  of  K-Web  tool  use,  successful  task  performance  means  providing  K-Web  users  an 
up-to-date  and  accurate  view  of  the  operational  situation. 

4.4.  KNOWLEDGE  WALL  VIEWER  SOFTWARE 

The  KWV  operators  at  both  Global  2000  and  Global  2001  indicated  that  they  liked  the  software, 
especially  with  regard  to  the  ease  with  which  it  allowed  them  to  move  information  around  on  the 
displays.  However,  problems  occurred  on  both  the  Global  2000  and  2001  K-Walls  when  multiple 
users  attempted  to  control  the  KWV  simultaneously  using  multiple  concurrent  inputs  (multiple  mice 
and  keyboards  and  touch  screen  input)  available  on  the  K-Walls. 
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The  Overview  Page  provided  on  the  KWV  included  functionality  allowing  users  to  easily  access 
the  Summary  Pages  and  move  them  to  other  windows.  Users  frequently  took  advantage  of  this  func¬ 
tionality.  The  color-coded  status  bar  on  the  Overview  Page  was  designed  to  give  consumers  an  inte¬ 
grated  view  of  the  operational  status  (by  showing  overall  status  over  time  for  all  the  operational  cells) 
and  whether  this  status  information  was  new  or  changed  (indicated  by  change  diamonds).  The  data 
indicated  that  users  did  access  pages  with  degraded  status  more  quickly  than  pages  with  unchanged 
or  improved  status,  suggesting  that  they  took  advantage  of  the  status  information  being  conveyed  on 
the  status  bar.  However,  users  did  not  access  Summary  Pages  indicating  a  change  related  to  status 
more  quickly  than  those  with  no  change  diamonds.  This  latter  result  suggests  that  better  change 
alerting  functionality  is  necessary. 

4.5.  TACGRAPH 

One  other  notable  difference  between  the  Global  2000  and  2001  war  games  was  in  the  use  of  Tac- 
Graph.  The  TacGraph  software  that  emerged  as  a  very  popular  and  useful  tool  at  Global  2000  was 
used  minimally  during  the  2001  game.  Based  on  observations  during  the  2001  game,  one  likely  rea¬ 
son  for  this  shift  is  the  use  of  the  tactical  software,  Battlescape  NT,  which  was  very  strongly  encour¬ 
aged  by  the  NWC  staff,  to  the  exclusion  of  other  map-based  applications.  Poor  TacGraph  utilization 
may  also  be  attributable  to  the  limited  training  provided  on  this  tool.  Most  players  were  not  aware  of 
TacGraph' s  capabilities  or  even  that  it  existed  for  use  in  the  game.  Comments  made  by  the  few  play¬ 
ers  that  used  TacGraph  can  be  found  in  Appendix  B. 

4.6.  TRAINING 

Many  of  the  differences  described  above  are  to  be  expected  given  the  many  factors  that  varied 
across  the  two  games.  These  include  organizational  variables  (differences  across  the  commands  that 
participated  in  the  game  and  across  the  organizational  structures  imposed  by  the  game),  differences 
in  the  scenarios,  and  differences  in  the  available  information  tools.  However,  the  most  notable  cause 
of  many  of  the  observed  differences  in  K-Web  use  between  the  two  games  was  the  difference  in  the 
amount  of  training  players  received  on  the  K-Web  tools  before  and  during  the  game. 

For  the  Global  2000  war  game,  users  of  the  K-Web  tools  and  products  (i.e.,  senior  decision-makers 
and  software  tool  users)  were  identified  far  in  advance  by  CJTF  staff  and  War  College  personnel. 
Identified  personnel  were  thoroughly  familiarized  with  the  intended  Concept  of  Operations  and  were 
given  adequate  training  and  extensive  practice  with  the  provided  K-Web  tools.  Specifically,  most 
users  received  approximately  1  to  2  hours  of  training  and  hands-on  practice  before  game-play  began, 
after  which  they  received  additional  one-on-one  assistance,  as  needed  during  the  first  few  hours  of 
use.  Due  to  a  number  of  factors,  the  training  conducted  at  Global  2001  was  limited  and  sporadic,  at 
best. 

The  importance  of  implementing  K-Web  training  sessions  and  training  tools  cannot  be  emphasized 
enough.  K-Web  products  and  tools  are  designed  to  be  as  user-friendly  and  intuitive  as  possible.  Nev¬ 
ertheless,  both  information  producers  and  consumers  require  training  in  order  to  use  the  K-Web  to 
their  greatest  advantage.  In  order  to  do  so,  K-Web  users  need  to  know  what  the  system  capabilities 
are  and  how  to  use  them.  They  also  need  to  be  familiar  with,  and  practice/adhere  to  the  business 
rules.  The  key  to  understanding  both  the  technology  and  the  business  rules  is  training.  Training  can 
play  a  critical  role  in  users’  motivation  to  utilize  the  system.  Needs  assessment  should  occur  before 
the  training,  in  order  to  identify  the  skill/knowledge  levels  of  the  participants,  as  well  as  their  under¬ 
standing  of  the  purpose  and  value  of  Global  war  games  and  related  technologies.  The  more  that 
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K-Web  users  can  make  a  connection  between  the  capabilities  of  the  system  and  the  requirements  of 
their  jobs,  the  more  likely  they  are  to  benefit  from  training  and  use  the  system  as  it  was  designed  to 
be  used  (Goldstein,  1993). 
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5.  CONCLUSIONS  AND  RECOMMENDATIONS 


Several  design  changes  were  made  to  the  K-Web  products  and  tools  based  on  the  Global  2000  use, 
utility,  and  usability  evaluations.  A  re-evaluation  of  the  K-Web  at  the  Global  2001  war  game  pro¬ 
vided  invaluable  insights  regarding  the  effectiveness  of  these  design  changes.  Overall,  the  K-Web 
proved  to  be  quite  successful  as  a  knowledge  management  and  SA  tool.  The  redesigned  Summary 
Pages  provided  users  with  an  easy-to-use,  standardized  view  of  information  contained  in  and  dis¬ 
seminated  via  the  K-Web.  The  change  alerting  functionality  provided  by  the  Summary  Pages  and 
Overview  Page  was  a  good  first  step  in  supporting  users’  attention  to  management  needs.  The  K- 
Web  display  tools  (KWVs)  and  other  support  tools,  such  as  the  SumMaker  and  KWV  software 
applications,  were  also  considered  highly  successful. 

In  addition  to  enabling  an  assessment  of  newly  developed  K-Web  technologies,  the  data  and 
observations  collected  during  the  Global  2001  war  game  point  to  specific  areas  that  can  be  improved 
by  future  K-Web  research,  development,  test,  and  evaluation  efforts: 

•  Develop  mechanisms  for  improved  dissemination  of  K-Web  business  rules  to  support  opti¬ 
mal  use  of  the  K-Web  by  information  providers  and  consumers. 

•  Continue  development  of  cognitive  tools  to  support  attention  management  and  change  detec¬ 
tion,  including  those  that  tailor  change  alerting  to  meet  the  needs  of  different  users. 

•  Research  means  of  providing  feedback  between  information  consumers  and  providers  related 
to  both  the  use  and  needed  content  of  K-Web  information  products. 

•  Develop  integrated  training  to  promote  a  better  understanding  of  K-Web  tool  capabilities  by 
information  producers  and  consumers. 
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APPENDIX  A: 

DATA  COLLECTED  AUTOMATICALLY  DURING  GLOBAL  2001 


Table  1 .  The  number  of  Summary  Pages  accessed  via  the  KWV 
software  on  the  K-Wall  and  K-Desks  (Figure  10). 


Number  of  Summary  Pages  Accessed/Hour 

PHASE3 

Average  (All  Global  2001  KWVs) 

30.0 

38.8 

CINC  K-Desk 

57.47 

70.18 

FCC  K-Desk 

25.73 

37.73 

CJTF  K-Wall 

9.51 

6.67 

8.55 

Global  2000  K-Wall 

22.3 

19.4 

43.0 

Table  2.  The  number  of  links  from  the  Summary  Pages  accessed 
via  the  KWV  software  on  the  K-Wall  and  K-Desks  (Figure  11). 


Number  of  Summary  Pages  Accessed/Hour 

PHASE1 

PHASE2 

PHASE3 

Average  (All  Global  2001  KWVs) 

41.46 

25.02 

27.18 

CINC  K-Desk 

55.03 

EEB 

FCC  K-Desk 

7.51 

msm 

CJTF  K-Wall 

61.84 

12.13 

Global  2000  K-Wall 

11.4 

12.3 

26.3 

Table  3.  The  number  of  links  included  on  the 
Summary  Pages  (Figure  12). 


Average  Number  of  Links/Summary  Page 

PHASE1 

PHASE2 

PHASE3 

FCC1 

FCC2 

8.48 

KEB 

5.93 

8.25 

FCC4 

5.85 

10.43 

3.93 

11.00 

CJTF 

6.13 

CINC 

Supp.  Area  1 

Supp.  Area  2 

4.24 

7.50 

11.43 

Mean 

5.75 

10.85 

10.03 

A-1 


Table  4.  The  format  of  links  included  on  the 
Summary  Pages  (Figure  13). 


Number  of  Links 

PHASE1 

PHASE2 

PHASE3 

MS  PowerPoint 

568 

429 

497 

MS  Word 

219 

89 

136 

MS  Excel 

0 

0 

19 

Graphic 

159 

119 

28 

HTML 

1283 

1287 

859 

Other 

0 

15 

37 

Table  5.  The  rate  at  which  Summary  Pages  were 
published  (number  of  Summary  Pages  posted  per 
hour  of  game  play)  (Figure  14). 


Summary  Pages  Published/Hour 

PHASE1 

PHASE2 

PHASE3 

FCC1 

3.05 

2.67 

1.64 

FCC2 

3.37 

5.20 

2.27 

FCC3 

0.74 

0.53 

1.36 

FCC4 

3.16 

1.87 

1.36 

0.79 

0.40 

0.27 

CJTF 

2.74 

2.13 

1.64 

CINC 

3.53 

7.20 

3.36 

Supp.  Area  1 

0.74 

0.40 

0.91 

Supp.  Area  2 

3.47 

2.93 

Mean 

2.40 

2.59 

1.64 

A-2 


Table  6.  The  percentage  of  Summary  Pages  that 
contained  one  or  more  change  diamonds  (Figure  15). 


%  of  Summary  Pages  With  Changes  Diamonds  Posted 

PHASE1 

PHASE2 

PHASE3 

FCC1 

17.24 

55.00 

100.00 

51.56 

100.00 

FCC3 

57.14 

50.00 

33.33 

FCC4 

26.67 

FCC5 

46.67 

100.00 

CJTF 

77.78 

CINC 

88.89 

Supp.  Area  1 

0.00 

70.00 

Supp.  Area  2 

40.91 

81.82 

100.00 

Mean 

34.96 

70.42 

77.44 

Table  7.  Activity  before  and  after  critical  events,  as  a  function  of  level 
of  criticality  (Figure  16). 


Actions/Minute 

Least 

Critical 

Moderately  Critical 

Most  Critical 

Before 

7.05 

5.7 

4.03 

After 

3.67 

3.6 

3.99 

Table  8.  The  format  of  the  information  displayed  on  Windows  2 
and  3  of  the  K-Wall  (Figure  18). 


%  of  Time  displayed 

PHASE1 

PHASE2 

PHASE3 

Summary  Page 

5.04 

3.05 

6.28 

Auxiliary  K-Web  Product 

64.74 

44.15 

40.39 

War  College  Web  Content 

26.92 

44.40 

48.34 

MS  PowerPoint 

3.21 

8.27 

4.84 

A-3 


Table  9.  The  format  of  the  information  displayed 
on  the  CINC  K-Desk  (Figure  19). 


%  of  Time  displayed 

PHASE1 

PHASE2 

PHASE3 

SP 

25.27 

21.84 

24.37 

Aux.  KWeb 

10.50 

OvView 

12.04 

NWC  Web 

13.36 

11.76 

MS  Off. 

4.22 

1.68 

Chat/Email 

11.06 

Tact. 

0.00 

0.14 

SmMkr 

TcGrph 

5.53 

1.31 

0.14 

Table  10.  The  format  of  the  information  displayed 
on  the  FCC  K-Desk  (Figure  20). 


%  of  Time  displayed 

PHASE1 

PHASE2 

PHASE3 

SP 

15.18 

12.29 

13.99 

Aux.  KWeb 

2.50 

4.46 

OvView 

12.68 

0.00 

0.30 

BEM 

Chat/Email 

Tact. 

7.08 

n 

SmMkr 

13.13 

TcGrph 

3.70 

0.63 

0.89 

Table  1 1 .  Content  of  the  information  displayed  on 
Windows  2  and  3  the  K-Wall  (Figure  21). 


%  of  Time  displayed 

PHASE1 

PHASE2 

PHASE3 

Comp.  Cdr 

5.22 

1.76 

7.64 

SuppArea 

41.24 

46.88 

39.68 

CJTF,  CINC,  HA 

6.98 

4.34 

CmdNet 

44.39 

48.34 

SitSum 

0.00 

0.00 

0.00 

TAPS 

0.15 

0.00 

0.00 

A-4 


Table  12.  Content  of  the  information  displayed  on 
the  CINC  K-Desk  (Figure  22). 


%  of  Time  displayed 

PHASE1 

PHASE2 

PHASE3 

37.62 

33.13 

7.47 

9.38 

CJTF,  CINC,  HA 

40.31 

38.25 

37.44 

Cdr.  Sum. 

1.11 

0.00 

CmdNet 

0.00 

0.00 

8.72 

SitSum 

0.00 

0.00 

TAPS 

21.06 

16.67 

9.37 

Table  13.  Content  of  the  information  displayed  on 
the  FCC  K-Desk  (Figure  23). 


%  of  Time  displayed 

PHASE1 

PHASE2 

PHASE3 

Comp.  Cdr 

42.05 

28.25 

SuppArea 

1.24 

0.64 

msm 

CJTF,  CINC,  HA 

6.02 

0.00 

Cdr.  Sum. 

0.00 

CmdNet 

SitSum 

16.80 

33.33 

20.00 

TAPS 

0.00 

0.00 

0.00 

Table  14.  The  percentage  of  published  Summary  Pages  that 
contained  change  diamonds,  as  a  function  of  level  of  criticality 
(Figure  24). 


Percent  of  Published  Containing  Diamonds 

No  Critical  Event 

21.15 

Least  Critical 

44.00 

Moderately  Critical 

50.94 

Most  Critical 

53.85 

A-5 


Table  15.  The  use  of  the  KWV  operations  (left  and  right  mouse 
clicks)  on  the  Overview  Page  to  display  Summary  Pages  and 
move  them  to  windows  on  the  K-Wall  and  K-Desks  (Figure  25). 


Number  of  Uses/Hour 

PHASE1 

PHASE2 

PHASE3 

CJTF  K-Desk  (left  mouse) 

13.51 

4.69 

11.36 

CJTK  K-Desk  (right  mouse) 

10.59 

9.27 

CINC  K-Desk  (left  mouse) 

7.62 

5.82 

CINC  K-Desk  (right  mouse) 

3.78 

0.46 

0.27 

FCC  K-Desk  (left  mouse) 

1.46 

0.08 

0.00 

FCC  K-Desk  (right  mouse) 

0.05 

0.00 

0.00 

Table  16.  Access  time  (by  the  CJTF  KWV)  for  Summary  Pages  as  a 
function  of  type  of  change  in  status  over  time  (Figure  26). 


Difference  Between  Time  Summary  Page  is  Published  and  Time  Summary 
Page  is  Accessed  on  JCC  K-Wall  (minutes) 

PHASE1 

PHASE2 

PHASE3 

SPs  Showing  Degraded  Status 

4.25 

12.70 

SPs  Showing  Improved  Status 

21.35 

10.15 

IFiiilB 

SPs  Showing  Unchanged  Status 

52.47 

49.58 

32.68 

Table  17.  Access  time  (by  the  CJTF  KWV)  for  Summary  Pages  as  a 
function  of  the  presence/absence  of  change  diamond  (Figure  27). 


Difference  Between  Time  Summary  Page  is  Published  and  Time 

Summary  Page  is  Accessed  on  JCC  K-Wall  (minutes) 

PHASE1 

PHASE2 

PHASE3 

All  SPs 

47.32 

25.15 

20.75 

SPs  with  Diamond 

56.47 

21.43 

26.07 

SPs  with  No  Diamond 

40.85 

37.28 

12.32 

Table  18:  Use  of  the  KWV  software  operations  on 
the  CJTF  K-Wall  (Figure  31). 


Uses/Hour 

PHASE1 

PHASE2 

PHASE3 

Left  Mouse 

13.51 

4.69 

11.36 

Right  Mouse 

■EBB 

3.85 

9.27 

Refresh 

0.00 

Channels 

0.00 

0.27 

Preset  1 

0.08 

0.09 

Show 

0.00 

0.00 

Hide 

0.09 

A-6 


Table  19:  Use  of  the  KWV software  operations  on 
the  CINC  K-Desk  (Figure  32). 


Uses/Hour 

PHASE1 

PHASE2 

PHASE3 

Left  Mouse 

0.77 

Right  Mouse 

| 

Refresh 

7.08 

Channels 

6.59 

Preset  1 

0.65 

0.00 

Show 

0.05 

Hide 

0.43 

0.23 

0.55 

Table  20:  Use  of  the  KWV  software  operations  on 
the  FCC  K-Desk  (Figure  33). 


Uses/Hour 

PHASE1 

PHASE2 

PHASE3 

Left  Mouse 

1.08 

1.88 

Right  Mouse 

1 

1.24 

Refresh 

1.30 

Channels 

2.00 

Preset  1 

0.22 

0.03 

Show 

0.72 

0.41 

0.12 

Hide 

1.59 

0.59 

0.42 

Table  21:  Format  of  information  displayed  on  the  Global  2000  and  Global 
2001  K-Walls  (Figure  34). 


APPENDIX  B: 

OBSERVATIONAL  DATA  COLLECTED  DURING  GLOBAL  2001 


Use,  Utility,  and  Usability  of  K-Web  Products  During  Global  2001 

Summary  Page  Use 

•  The  Commanders  ’  Summaries  are  being  accessed  by  the  cell  players  and  projected  up  on  the 
K-Wall. 

•  [FCC2]  uses  the  K-Desk  to  get  SAfrom  the  other  Summary  Pages... so  the  KM  can  keep  track 
of  other  pages  and  to  update  his  own  Summary  Page  and  documents. 

•  [FCC1  ]  used  the  K-Desk  to  pull  up  Commanders  ’  Summaries  and  see  what  others  were 
doing. . . 

•  I’m  seeing  guys  dipping  down,  grabbing  pictures,  graphics,  text  and  providing  it  to 
Commanders. 

•  In  my  cell,  5-6 people  at  a  time  are  looking  at  the  Commanders’  Summary  Pages. 

•  I’m  looking  at  the  pages  to  make  sure  no  information  needs  filling  in.. .there’s  good 
information  that’s  been  converted  to  knowledge  if  you  know  where  to  get  it. 

•  The  Commanders  ’  Summary  Page  -  gave  [FCC2 ]  a  way  to  give  picture  to  the  JTF  staff  the 
way  that  we  wanted/thought  they  should  see  it. 

•  When  I  wasn ’t  posting,  I  was  trolling  other’s  Web  pages.  ..to  be  apprised  of  others. 

Number  and  Format  of  Summary  Page  Links 

•  BWC  is  telling  others  how  their  information  will  be  a  link  from  the  CJTF  Summary  Page. 

•  [. .  .looking  at  a  link  on  the  K-Wall]  Can  we  use  this  in  our  brief? 

•  [FCC1]  indicates  that  their  information  for  evening  brief  is  in  link  from  Summary  Page. 

•  [FCC2 ]  uses  K-Desk  to  get  SAfrom  other  Summary  Pages,  for  KM  to  keep  track  of  other 
pages  and  to  update  their  own  Summary  Page  and  documents. 

•  Ask  [FCC1  ]  what  all  these  circles  are  ?  Put  it  on  your  slide. 

•  The  CINC  [ Summary  Page ]  has  a  link  to  ROEs,  can  you  put  it  on  our  Summary  Page  ? 

•  The  CINC  page  has  a  link  to  [SuppAreal]  links  (same  that’s  always  displayed  on  W2) 

Summary  Page  Update  Rate 

•  [ FCC5 ]  has  got  their  stuff  populated. . .  that ’s  good. 

•  I’m  seeing  guys  dipping  down,  grabbing  pictures,  graphics,  text  and  providing  it  to 
Commanders. 

•  There  still  isn  ’t  enough  content  on  the  Summary  Pages. 

•  We've  looked  on  [FCC3]  and  [FCC2]  pages  and  can’t  find  evidence  of  [trigger]. ..We’ve 
already  sent  them  a  message  to  update  their  Commanders  ’  Summary  Pages. 

•  The  information  is  coming  too  fast  to  update  the  slide  (link  re:  above)  anymore... we  were 
told  to  use  the  tools  until  they  break  and  it  looks  like  they  have. 
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•  [CJTF  staff  to  component  commands]:  We  need  all  of  you  to  update  your  Commanders'  Sum 
pages  with  information  about  where  you  are . . . 

•  E-mail  from  HA  to  [information  provider]  to  update  their  page. .  .HA  is  not  getting  the  picture 
they  need. 

•  We  need  you  to  update  the  Commanders  '  Summaries  every  morning ,  afternoon  and  as 
needed  for  every  one.,. need  to  push  other  information  during  the  day. 

•  We  used  the  Summary  Page  to  post . . .  data  [ that]  wasn  7  used  much  in  the  game  so  we  didn  7 
update  much  (daily). 

Summary  Page  Preference 

•  The  Summary  Page  gave  [FCC2]  a  way  to  give  picture  to  the  JTF  staff  the  way  that  we 
wanted/thought  they  should  see  it. 

•  It  standardized  everyone ’s  input. 

•  We're  trying  to  figure  out  templates  for  our  Web  Page  on  the  ship... these  Summary  Pages 
would  work. 

•  The  tools  found  to  be  very  useful  included  the  K-Wall  setup ,  Commanders'  Summaries... 

New  Summary  Page  Design 

Change  Alerting 

•  The  diamonds  were  good. . . 

•  The  main  feature  that  is  needed  is  alert  functionality. 

•  How  do  you  alert,  tailor  information  for  different  CCs? 

•  You  can  post  things  on  the  Web  all  day  long  but  you  need  to  push  information... can't  assume 
people  will  pull  off  Web. 

•  Need  an  indication  of  when  alerts  were  changed  or  added. 

•  Since  it's  always  on  the  display... if  you  know  in  advance  what  you  want  to  be  alerted  to  you 
could  be  automatically  alerted... it  took  too  long  to  find  these  things  out. 

•  Is  there  any  way  we  can  make  a  history  of  these  pages? 

•  There  are  still  problems  that  purple  links  aren’t  accessed  because  they  are  thought  to  go  to 
old  information. 

Other  Features  and  New  Layout  of  Summary  Pages 

Summary  Pages 

•  I  like  the  idea  of  everyone  having  information  in  the  same  format. 

•  I  need  more  space  to  put  stuff. 

•  [The  Summary  Page]  is  a  good  looking  page  with  enough  white  space. 

•  Need  ability  to  scroll  or  add  space. 

•  There  was  only  a  limited  amount  of  information  you  could  put  on  it. 

•  The  Summary  Pages  were  too  simplistic. 
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•  It  wasn’t  flexible  enough.. .you  couldn’t  change  the  information. 

•  You  should  always  put  military  time  on  the  Summary  Pages. 

•  Nice  feature  to  have  the  time  stamp  though. 

Miscellaneous  Links  Page 

•  The  font  size  for  Miscellaneous  Links  is  too  large. 

•  Change  the  template  for  Miscellaneous  Links  so  you  can  include  graphics  there.  ..easier  to 
look  at  than  text.  Make  it  closer  to  the  [Summary  Page]... 

•  Users  need  to  be  able  to  add  diamonds  and  squares  to  Miscellaneous  Links. 

Use,  Utility,  and  Usability  of  the  K-Wall/K-Desks  During  Global  2001 
Format  and  Content  of  Information  on  the  K-Wall/K-Desks 

•  It  would  be  better  have  CNN  up  there. ..and  information  about  Intel  and  Effects. 

•  The  CJTF  staff  has  requested  a  Summary  Page  from  FCC5  as  well  as  access  to  it  via  a  button 
on  the  Overview  Page. 

•  The  CJTF  staff  has  requested  a  Summary  Page  from  SuppAreal  as  well  as  access  to  it  via  a 
button  on  the  Overview  Page. 

•  The  CJTF  staff  has  requested  a  Summary  Page  from  SuppArea2  as  well  as  access  to  it  via  a 
button  on  the  Overview  Page. 

•  The  morning  CJTF  brief  is  presented  on  the  K-Wall. 

•  Having  the  [K-Desk]  screen  and  the  COP  on  the  shared  display  really  helps. 

K-Wall  and  K-Desk  Use 
K-Wall  Use 

•  People  in  the  JCC  completely  relied  on  the  K-Wall. 

•  The  K-Wall  has  been  really  good  ...we  can  put  whatever  we  want  up  there. . .  The  BWCs  are 
on  that  part  of  it... it’s  good  for  situation  awareness. 

•  I’m  just  using  it  as  my  own  personal  computer  since  I  don ’t  have  one. 

•  In  the  JCC,  besides  the  Knowledge  Wall,  there  was  no  management  of  information. 

•  Most  activity  the  last  hour  revolves  around  prepping  for  afternoon  brief. 

•  BW1  announces  “[SuppAreal]  update”  or  details  of  change  whenever  [SuppAreal]  has  new 
information.... 

•  [Three  CJTF  staff]  are  discussing  information  on  K-Wall. 

•  All  staff  looking  at  K-Wall. .  .working  on  the  brief. 

K-Desk  Use 

•  The  Knowledge  Warrior  maintained  the  K-Wall/K-Desk  and  this  was  different,  inconsistent 
across  different  cells. 
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•  The  difference  in  how  the  K-Wall  was  used  shows  how  it's  driven  by  the  human 
element. .  .whatever  we  put  out  there  has  to  be  flexible  enough  to  deal  with  human  element 
(configuration  differences ,  preference  for  text  or  graphics  etc...) 

•  In  [FCC2],  we've  projected  two  screens  from  each  K-Desk  onto  the  large  screen  display.  In 
[FCC3],  2  screens  from  K-Desk  are  on  large  screen  display. 

•  [FCC2]  uses  K-Desk  to  get  SAfrom  other  Summary  Pages ,  for  the  KM  to  keep  track  of  other 
pages  and  to  update  their  own  Summary  Page  and  documents. 

•  The  Commander  didn  7  look  at  the  K-Desk  because  he  was  always  in  chat  rooms 

•  We  used  the  [ K-Desk ]  on  watch  a  lot... 

•  In  some  cells  they  had  useful  info  on  shared  displays. ..others  didn 7  work  so  well. 

K-Wall/K-Desk  Preference 

•  The  K-Wall  has  been  really  good  ...we  can  put  whatever  we  want  up  there... 

•  I  like  the  capability  to  put  any  window  where  you  want  it. 

•  /  like  the  K-Wall  (K-Web?)  approach  because  it  uses  HTML  -  shared  files. 

•  Excellent  tool! 

•  It' s  a  great  tool! 

•  We  need  this  [=touch  screen  capability]  on  the  ship  for  the  admiral's  brief. 

•  I  think  the  only  thing  I  like  about  it  [the  K-Wall]  is  the  touch  screens. 

•  The  K-Wall  and  K-Desk  were  highest  ranking  tools  in  the  FCC2  survey  of  tools 

•  I  liked  K-Desk  set  up,  ability  to  view  multiple  images 

•  The  K-Wall  helped  us. ..the  BWC  was  comfortable  using  it. . . 

•  Any  tool  like  this  should  communicate  information  to  the  boss,  to  peers,  to  subordinates. .  .the 
K-Wall  had  flexibility  to  do  that. 

New  KWV  Design 

New  Overview  Page 

•  Things  are  turning  blue,  telling  me  that  there  are  changes  so  I'm  going  to  check  them. 

•  [SuppAreal]  IP  shows  BW1  his  page  -  alerting  by  him  (via  coming  up  to  command  table  to 
let  him  know  that  he’s  updated)  It's  faster  than  blue  text  on  viewer. 

•  It's  hard  to  see  that  button  text  (on  Viewer)  has  changed  to  blue  from  a  distance... maybe 
change  to  red  or  some  other  color. 

•  They  have  blue  stuff  but  no  change  diamonds. 

•  /  recommend  that  when  changes  are  made  to  Commander  Summaries  that  “ diamond  " 
flash/pulse  for  15-20  secs  in  order  to  alert  people  immediately  of  new  changes. 

•  Need  something  that  pulses  for  ~20  seconds  when  something  changes  to  a  diamond.  ..need  to 
know  that  information  has  changed.  Blue  text  is  not  an  “ alert"  -  need  something  more  salient 
(like  on  WIGS  but  not  flashing  all  the  time). 
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•  I  wish  they  had  status  for  Alerts  &  Impacts  on  K-Wall...we  changed  Today,  Tomorrow,  LR 
status  once  a  day  but  changed  Alerts  more  often. 

•  It  should  indicate  on  Summary  Page  when  you’ve  updated  a  linked  document,  e.g.,  change 
the  Today  link  to  a  diamond. 

•  It  was  easy  to  just  make  changes  to  files... without  having  to  republish  everything. 

•  You  can  post  things  on  the  Web  all  day  long  but  you  need  to  push  information... can’t  assume 
people  will  pull  off  Web. 

Overview  Page  General 

•  It’s  easy  to  use. 

•  The  Overview  Page/K-Web  could  almost  do  away  with  voice  communications. 

•  The  Web  page  ( Overview  Page  on  WIGS)  was  not  intuitive  because  it  didn’t  match  the 
Overview  Page  on  the  KWV. 

Number  of  Monitors 

Visibility 

•  It ’s  hard  to  see  that  button  text  ( on  Viewer)  has  changed  to  blue  from  a  distance. .  .maybe 
change  to  red  or  some  other  color. 

•  Problem  with  people  standing  in  front  ofK-Wall  -  the  KW-operator  ( J-KW1 )  can ’t  see  K- 
Wall  (trying). 

•  Can  I  move  the  Select  Monitor  window  to  another  screen?  [because  people  are  standing  in 
front  of  it]. 

•  As  soon  as  we  can  get  a  clear  view  of  the  screen,  we’ll  pull  it  up. 

•  There  should  be  no  walkway  in  front  of  it... that’s  been  a  problem  a  couple  times  today. 

•  The  K-Desk  needs  to  be  big  enough  for  other  people  to  see  it. 

•  It’s  hard  to  read  that  [MS  Word  document ]. 

•  Can  I  send  them  and  e-mail  complaining  about  their  [link]  ...  It’s  hard  to  read  it. 


K-Wall 

•  There  aren’t  enough  screens  to  show  everything  ( looking  at  PowerPoint  brief  from  [FCC2]) 

•  3  screens  aren  ’t  enough. .  .probably  need  4  or  5... 3  might  be  enough  if  you  could  use  the  1st 
one.  They  want  Ops  up  here,  Intel  on  there,  they  want  the  logs... 

Overview  Page 

•  This  stuff  [on  W1  ]  isn’t  useful  because  it  takes  1/3  of  the  K-Wall... don’t  need  the  key  down  in 
the  comer.  Better  to  have  tabs  at  top  of  screen  showing  status  for  each  area,  that  you  could 
click  to  get  that  area.  You  could  fit  up  to  14.  You  wouldn’t  need  to  right  click  to  move  into 
another  window. 

•  I  usually  had  just  one  window  open... the  Overview  screen  took  up  too  much  space. 

•  I  like  the  Overview,  but  it  only  let’s  you  see  half  of  a  page. 
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K-Desks 

•  I  liked  the  K-Desk  set  up ,  ability  to  view  multiple  images. 

K-Web  Support  Tools 

SumMaker  Software  Usability  and  Features 
SumMaker  Usability 

•  It  made  it  easy  to  produce  Web  pages  with  minimal  training. 

•  I  wish  [other  tools]  were  as  obvious...  wish  it  were  all  this  easy. 

•  I  had  no  trouble  learning  to  use  it. 

•  It  was  easy  to  use  once  somebody  showed  you  how  to  use  it. 

•  It’s  easy  to  get  information  in  there. 

•  It  looks  very  sailor  proof.  I  could  take  a  junior  person  and  they  would  be  very  comfortable  in 
10-15  minutes. 

•  Simplicity ...  really  easy  to  use.  Easy  to  link  files. 

•  I  only  needed  15  minutes  to  do  training. 

•  Ids  easy  to  get  information  in  there. 

•  I  only  got  4-5  minutes  of  training. .. within  a  half  hour  most  computer  literate  people  could 
use  it  too. 

•  I  never  published  a  Web  page  before  I  got  here. 

•  Eve  never  done  any  Web  publishing  before. 

•  Ids  very  intuitive ,  user-friendly... it  required  minimal  training. 

•  It  had  a  clean  Interface. ..simple  and  intuitive. 

•  Miscellaneous  Links  took  a  lot  of  steps  ( problem  with  finding  Miscellaneous  Links  saved  file). 

•  The  Miscellaneous  Links  page  was  cumbersome... too  restrictive  because  it  was  a  list  ...you 
couldn't  group,  organize  [need formatting  tools]. 

•  The  transition  between  SumMaker  and  Miscellaneous  Links  interface  was  not  obvious. 

•  I  couldn  yt  go  back  to  SM  when  in  Miscellaneous  Links. 

•  It  was  confusing  that  you  had  to  copy  files  into  directory  -  step  takes  it  away  from  the  user. 

•  It  was  hard  for  people  to  remember  to  put  stuff  in  the  [  right]  folder. 

•  Moving  information  from  on  place  to  another  is  what  slows  things  down.  ..why  ids  not 
working  in  some  cells. 

•  There  was  a  problem  when  people  wanted  to  use  information  that  they  saw  on  a  Summary 
Page  link  in  slides  but  it  had  been  changed  since  they  saw  it  last. . . 

•  Users  need  one  button  to  publish  and  save,  keep  SM  open. . . 

•  Too  many  mouse  clicks  to  publish  it. 

•  Ids  a  problem  that  you  have  to  click  so  many  buttons  when  . ..not  a  big  problem  but  you  want 
to  instantly  publish  things  once. 
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•  It  would  have  been  nice  if  the  computer  clock  had  been  set  to  game  time .  It  wasn't  that  easy 
to  add  the  time... nice  feature  to  have  the  time  stamp  though. 

•  One  problem  was  having  to  put  in  game  time... but  this  wouldn't  be  a  problem  in  the  real 
world. 

•  You  had  to  remember  to  change  the  time/date  stamp. .  .you  need  the  flexibility  so  that  you 
have  the  choice  to  get  system  time  from  simulation  or  real  time. 

SumMaker  Features  and  New  Requirements 

•  It  was  a  useful  way  to  present  information  from  [SuppArea2]. 

•  It  was  a  good  program. .  .All  in  all  I  don 't  have  any  complaints. 

•  The  capability  was  good. 

•  I  didn't  get  near  to  taking  advantage  of  its  capability/discovering  its  limitations.  For  what  it 
was  designed  to  do>  it  was  fine. 

•  [ SumMaker]  pretty  much  fulfilled  the  requirements. . . 

•  Nice  to  have  a  template ,  to  be  able  to  keep  updating... 

•  It  standardized  everyone's  input. 

•  I  like  the  idea  of  everyone  have  information  in  the  same  format. 

•  The  flexibility  in  linking  was  good. 

•  There  was  a  loss  of  flexibility  ( because  of  the  same  reasons  that  make  it  sailor-proof). 

•  It  supported  MS  Office  products  and  file  formats. 

•  The  flexibility  in  linking  was  good. 

•  We  don't  stop  to  prepare  briefs... information  in  briefs  is  information  we're  already  using  for 
own  operations. 

•  We  used  PowerPoint  to  make  sub  pages. 

•  There  was  a  loss  of  flexibility  ( because  of  the  same  reasons  that  make  it  sailor-proof). 

•  I  liked  the  fact  that  you  could  drop  information. 

•  Users  need  the  ability  to  archive  old  information. 

•  It  was  hard  to  archive . . . 

•  It  would  be  nice  to  be  able  to  archive  old  information. 

•  There  was  a  problem  when  people  wanted  to  use  information  that  they  saw  on  a  Summary 
Page  link  in  slides  but  it  had  been  changed  since  they  saw  it  last ... 

•  We  wanted  to  link  to  archived  the  Supp Areal  stuff... there  weren't  enough  links  to  do  that. 

•  A  phone  book  feature  on  Summary  Pages  would  be  nice  (mouse  over  for  details  about 
person ,  hyperlinks  to  each). 

•  You  might  want  a  link  to  another  collaboration  system  (e.g.,  whiteboard). 
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Feedback  to  Information  Providers 

Feedback  related  to  Summary  Page  access 

•  I didn ’t  know  how  my  Summary  Page  was  used. ..didn ’t  even  know  who  was  looking  at  it. 

•  You  need  to  have  feedback  that  people  are  using  your  information. 

•  Better  feedback  is  necessary. 

•  I  don’t  know  what  the  CINC  is  getting  from  us  [the  CJTF  staff ]  - 1  know  it’s  on  our  page  but 
I  don ’t  know  that  the  CINC’ s  looking  at  it. 

•  You  need  information  about  who ’s  viewed  your  information  (specifically  who)  so  you  can 
talk  to  the  person  about  it  -  even  if  in  form  of  e-mail. 

Feedback  about  Summary  Page  content 

•  We’ve  already  sent  them  a  message  to  update  their  command  Summary  Pages. 

•  We  need  all  of  you  to  update  your  Commanders’  Sum  pages  with  information  about  where 
you  are... [chat  message]. 

•  I’m  looking  at  the  pages  to  make  sure  no  information  needs  filling  in.  if  there  is,  I  tell  BW1 
and  he  lets  the  guy  know...BWJ  is  pretty  proactive  about  it  too. 

•  E-mail  from  CINC  that  Today’s  link  is  still  yesterday’s  plan. 

•  There’s  not  enough  time  in  the  game  for  the  CJTF  to  figure  out  what  he  wants,  at  what  point 
do  things  go  from  yellow  to  red.  It  needs  to  be  put  in  place  before  game. 

•  We  didn ’t  get  enough  feedback  from  upstairs  [JCC]  about  whether  there  was  right 
information. 

•  I  didn’t  feel  like  anyone  was  using  it.  If  they  were,  we  would  have  gotten  more  feedback  from 
other  people  about  what  they  needed  to  see. 

•  We  need  to  know  what  the  CJTF  wants  to  see. . . 

•  I  didn ’t  get  any  comments  about  what  the  Component  Commanders  wanted  to  see. 

New  KWV  Software:  Usability 

•  It  was  easy  to  display  information  on  the  K-Wall. 

•  The  K-Wall  helped  us. .  .BWC  was  comfortable  using  it. . . 

•  Can  I  move  the  Select  Monitor  window  to  another  screen  ? 

•  I  wish  there  was  an  easier  way  to  put  stuff  on  windows  other  than  right  clicking. 

•  Problem  that  you  didn ’t  know  where  a  window  would  pop  up. 

•  Problem  caused  when  one  user  is  trying  to  use  the  touch  screen  at  the  same  time  as  another  is 
using  the  mouse  to  control  the  K-Wall. 

•  Problems  when  we  wanted  to  run  a  PowerPoint  slide  show... when  we  clicked  on  another 
window,  slides  advanced  ( because  it’s  a  single  desktop)  ...you  should  be  able  to  keep 
functions  running  in  the  background. 
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Other  Emerging  Themes 

Training  and  Business  Rules 
Training 

•  The  training  was  good. 

•  The  training  documents  were  fairly  intuitive. ..didn’t  have  problems  figuring  things  out. 

•  It  would  have  been  nice  if  the  WIGS  training  had  let  people  know  the  Summary  Pages  were 
there. 

K-Web  Concept 

•  How  do  we  know  that  the  Component  Commanders  know  what’s  going  on? 

•  Does  the  CINC  have  access  to  the  same  information  as  we  do? 

•  We  need  help/advice  about  how  to  take  advantage  of  the  wall. 

•  At  the  beginning  of  the  game  you  don ’t  know  what  you  need  to  know. 

•  The  system  assumed  people  knew  where  to  go  for  information. 

•  We  didn’t  use  it  the  way  it  was  intended  to  be  used... The  learning  curve  started  slow,  we 
didn’t  know  how  were  going  to  use  it. 

•  There  should  have  been  business  rules  about  how  to  share  information... some  guys  used 
Commanders  Summaries,  some  used  shared  drives... 

•  Fusing,  focusing,  displaying  information  is  different  at  different  levels. 

•  There’s  not  enough  time  in  the  game  for  the  CJTF  to  figure  out  what  he  wants,  at  what  point 
do  things  go  from  yellow  to  red  needs  to  be  put  in  place  before  game. 

•  We  need  to  be  careful  about  standardization  because  we  also  used  it  as  our  means  of 
updating  ourselves. 

•  The  KWarrior  maintained  the  KW/K-Desk  and  this  was  different,  inconsistent  across 
different  cells. 

•  We  didn  ’t  have  an  information  management  or  organizational  plan  in  place. . . 

Specific  K-Web  Business  Rules 

•  We ’ve  fought  the  war  today  exactly  the  way  we  weren  ’t  supposed  to. .  .with  PowerPoint. 

•  E-mail  suggests  that  the  [information  providers]  move  some  Alerts  into  related  Information 
and  Links. 

•  They  used  diamonds  in  Alerts  and  Impacts  but  not  in  the  Today,  Tomorrow,  Long  Range 
area. 

•  There  has  been  a  lot  debate  regarding  content:  Where  [on  information  products]  should 
information  be  red  -  [ i.  e.  if  not  in  status  over  time]. 

•  I  didn’t  fell  like  ROE  for  use  of  system  was  established,  e.g.,  when  to  use  a  green  diamond  vs. 
a  square,  how  long  a  diamond  should  be  up... there  was  nothing  in  place  that  indicated  what 
should  be  pushed  or  pulled. .  .how  the  systems,  including  the  collaboration  tools,  should  be 
used. 
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•  The  Admiral  did  not  know  what  the  colors  meant. you  need  some  sort  of  key  to  the 
symbology. . . 

Comparison  to  last  year’s  K-Web 

•  An  example  showing  how  this  game  has  taken  a  step  backward  from  last  year's  Global  is  the 
morning  briefings. 

•  Last  year ;  the  K-Wall  was  updated  so  that  brief  didn  't  need  to  be  ready  by  1600  and  stop 
game  play.  It  was  a  different  way  of  doing  business. 

•  Last  year  the  BWC/CJTF  indicated  what  he  wanted  from  the  1st  day  of  the  game... what  was 
red,  what  was  important 

•  Last  year,  we  thought  the  K-Wall  was  great...  but  it  was  a  collection  of  information  that  was 
current  0  time  late... it  needed  projected  information ,  EBO  etc. 

•  CCG3  's  JTFEX  this  year  was  incredible.  ..because  they  had  experience  with  tools. . .and  the 
tools  were  tailored  for  their  needs. 

•  The  K-Wall  started  out  as  unwieldy... not  very  workable  product  but  [last  year's  players]  said 
“let  me  play  with  it  a  while  ”  and  it  turned  into  something  useful . 

TacGraph 

•  A  problem  with  TacGraph  is  that  text  size  does  not  change  when  you  zoom  in  or  out. 

•  When  you  click  on  the  Unit  tool  to  add  a  unit  but  change  your  mind,  can't  unselect  it... have 
to  add  a  unit  and  then  delete  it. 

•  Put  an  undo  feature  in  TacGraph. 

•  TacGraph  wasn't  very  useful... would  have  been  if  you  could  import  pics  from  other  tool. 

•  Tabbing  between  the  lat  and  long  didn't  work.  You  need  a  distance  tool... couldn't  specify 
dimensions  of  objects. 
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